Getting started with Open Data Kit and CentOS VPS

المشرف العام

Administrator
طاقم الإدارة
Open Data Kit or ODK is (probably) one of the most underrated survey data management platform. Yet ODK powers a lot of successful commercial platform, (for example CommCare) and helping tons of commercial and non-commercial activities across the globe. As the name suggests, Open Data Kit is open source and can be downloaded, customized, deployed and used with few easy steps. Let’s get started with setting Open Data Kit on CentOS VPS.



At first let us discuss what is Open Data Kit really mean. ODK is a collection of tool to collect and manage survey data using android mobile devices. It creates the survey form, uses the form to collect data from the handheld devices, stores and publishes the data as per the user’s need. ODK Build is a tool to create a form. ODK Collect uses the form to collect and send data to ODK Aggregate. ODK Aggregate receives the data and does various data management, including export into CSV and KML. Users can even use MS Excel or Open Office to create the form without writing a single code. This is all quite simple, read on for the steps.



ODK needs a place where you can deploy and customize your own Tomcat server and database. In order to achieve this you can either use a dedicated PC, a virtual machine or something like Docker, PaaS like App Engine or even a VPS. I tried the last one.



We will install JAVA into it.

sudo yum updatesudo yum clean allsudo yum search java | grep -i --color JDKsudo yum install java-1.8.0-openjdk.x86_64 java-1.8.0-openjdk-devel.x86_64java -versionThe clean command clears any cached files and folders. search and color searches available version of java and highlights them. Finally install installs them. The Java SDK package will be installed in /usr/lib/jvm directory by default. Use this to set JAVA_HOME temporarily. You can set the JAVA_HOME permanently by adding this line to “$HOME/.bashrc“.

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.x86_64/echo ${JAVA_HOME}SETTING UP TOMCAT

The latest stable Tomcat is version 8.5.6 and you can download that from the official website. The quickect way to install it to un-tar it to /opt/ and access it from there.

cd /home/sr/odksudo wget http://www-eu.apache.org/dist/tomcat/tomcat-8/v8.5.8/bin/apache-tomcat-8.5.8.tar.gzsudo tar -xvzf apache-tomcat-8.5.6.tar.gzsudo mv apache-tomcat-8.5.6 /opt/tomcatSome of you may wonder what are these codes doing. The wget is a tool to download anything to the folder /home/sr/odk which I already set (I am going to download everything in this folder). The tar command extracts the tar.gz archive. The mv stands for move (or cut and paste), I am moving the extracted folder to /opt/tomcat folder. Notice that you are not required to create those folders beforehand, they’ll create themselves (except the odk folder). Now see if you can start the tomcat service.

sudo /opt/tomcat/bin/catalina.sh startMake a quick check if your Tomcat installation works. In your browser visit the IP address to see a generic Apache page. Hit the xxx.xxx.xx.xx:8080 in your browser.



SETTING UP MYSQL

My VPS is quite cheap, only has 1GB of RAM. So I decided to shut down any non-essential services during boot. If you want to add Tomcat as a startup service use this script. We have Tomcat ready and running. Now we will need MySQL running, but you can also try PostgreSQL. To install MySQL we will need this,

sudo yum install mysql-server mysqlOn fresh installation of MySQL Server, the root user and password is blank. Setting a password here is a good security practice. To manage this use this command.

mysql_secure_installationSet a password. Go on with other settings with typing y.

Remove anonymous users? [Y/n] y
Disallow root login remotely? [Y/n] y
Remove test database and access to it? [Y/n] y
Reload privilege tables now? [Y/n] y

Done! Now we will check if our MySQL server works. Use your password to login.

mysql -u root -pODK Aggregate needs Java to run. ODK also needs a webserver (Tomcat) and a database (MySQL). Conventional web hosting services give you the later two – webserver and database, but do not give you the priviledge to install your own Java. ODK Aggregate can be installed in cloud platforms like Google and Amazon, private servers and VMs. We are using our own webserver built inside a VPS. The reason for that is there are a lot of people (means people like me) who cannot afford a server with secured and managed internet access for 24/7. If you have a short term survey work for a time sensitive project, you may not have enough resource to buy and maintain their own server and space to keep it. Even if you already have a server, you may not have time to configure it for a secured private IP, some IT folks to look after when you are sleeping, or even an internet connection with decent speed. So I decided to go for a cheap VPS which has all the above, minus the hassle of preconfiguration and the money issue. VPS can go down as much as $7/month. My VPS costs almost the same, has only 1GB RAM (pretty cheap, huh), 25GB disk space and a minimum internet speed of 100Mbps. I can always pay extra to add some RAM to my box, or can unsubscribe at my will. It’s an unmanaged CentOS 6 box with no cpenel. If you are not familiar with Linux environment, you can pay a few bux extra to install a cpenel. Cpenel or Control Penels can be used to manage files, install software and do all kinds of thing inside VPS with few clicks, you won’t need to learn any command. Some websites claim that it’s “extremely difficult” to manage a box without cpenel, but believe me it’s not that hard.



Without further ado, let’s install an ODK Aggregate.



SETTING UP AND CONFIGURE ODK AGGREGATE



First we will download the proper version of ODK Aggregate.

sudo wget https://s3.amazonaws.com/opendatakit.downloads/ODK%20Aggregate%20v1.4.12%20linux-x64-installer.runsudo chmod +x ODK\ Aggregate\ v1.4.12\ linux-x64-installer.runsudo ./ODK\ Aggregate\ v1.4.12\ linux-x64-installer.runYou will meet a long configuration process. You may not need to tweak all of these because most of the time the default value is just fine.



Here the installer will create a .war and .sql file based on the configurations you set. The .sql file will have the codes necessary for the databases and the .war file will set the web application. Keep pressing return (Enter) to scroll through the initial setup files, probably will take seven – nine returns.



And then,

Do you accept this license? [y/n]: yHere is everything you will meet while the installation.

---------------------------------------------------------------------------- Select an output parent directory This "installer" does not actually install anything. Instead, it guides youthrough configuring ODK Aggregate. As the last step of the installer, it willeither launch the upload tool for uploading to Google's AppEngine cloud servicesor open a README.html containing the final installation instructions for MySQLor PostgreSQL deployments. Please select the parent directory under which an 'ODK Aggregate' directory willbe created to contain the configured software. []: /home/sr/odk ----------------------------------------------------------------------------Choose Platform Choose the data storage and execution environment on which you would like ODK Aggregate to run. If you choose MySQL or PostgreSQL, you will need to first install that database server and an Apache Tomcat 6 webserver. [1] Google App Engine: Uses Google's cloud-based data storage and webserver services.[2] MySQL: Uses a MySQL database and an Apache Tomcat 6 webserver that you install.[3] PostgreSQL: Uses a PostgreSQL database and an Apache Tomcat 6 webserver that you install.Please choose an option [1] : 2 ----------------------------------------------------------------------------Pre-installation Requirements - Apache Tomcat Before continuing, you should download and install an Apache Tomcat 6 webserverwithin which you will run ODK Aggregate (instructions on installing your chosendatabase will be presented later). Apache Tomcat 6 can be downloaded from the link below. If you modify any of the Apache Tomcat defaults, please make note of them. http://tomcat.apache.org/download-60.cgi [Y/n]: y ----------------------------------------------------------------------------Apache Tomcat SSL configuration SSL certificates allow a client to verify the identity of a webserver and toestablish secured (encrypted) communications with that webserver (https:). SSLcertificates are typically purchased from Verisign or another issuing authority;that process can take weeks to complete. If you do have an SSL certificate, please refer to the Apache Tomcatdocumentation for how to install the certificate on your system. Within theTomcat 6.0 documentation, refer to the User Guide section 12: SSL. http://tomcat.apache.org/tomcat-6.0-doc/ssl-howto.html [Y/n]: y ----------------------------------------------------------------------------Apache Tomcat SSL configuration Do you have an SSL certificate? [1] No, I do not have an SSL certificate: - User logins ARE secure.- Submitting and/or viewing form data MAY NOT BE secure.- Changing passwords MAY NOT BE secure.Security breaches are particularly likely over unsecured WiFi hotspots. [2] Yes, I have an SSL certificate: - User logins ARE secure.- Submitting and/or viewing form data IS secure.- Changing passwords IS secure. Please choose an option [1] : 1 ----------------------------------------------------------------------------Apache Tomcat Port Configuration ODK Aggregate needs to know the port numbers and the Internet-visible IP addressor DNS name through which it can be reached in order to construct links back toits content when publishing or exporting data files. Follow the ODK Aggregate Tomcat Install instructions (link below) to make yourcomputer accessible on the web (using an Internet-visible IP address) and toestablish a DNS name. http://opendatakit.org/use/aggregate/tomcat-install/#Configure_for_Network_Access [Y/n]: y HTTP/1.1. Connector Port: [8080]: 8080 Internet-visible IP address or DNS name: [srv1.clubgis.net]: xxx.xxx.xx.xx ----------------------------------------------------------------------------Pre-installation Database Requirements - MySQL We have developed and tested against MySQL 5.1 and 5.5 Please download, install and configure MySQL from the link below. Choose the Downloads (GA) tab, and select the MySQL Community Server edition.When running the MySQL Server Instance Configuration Wizard, please make note ofyour root password and any non-default values you have entered during theinstallation process. http://www.mysql.com/downloads/ [Y/n]: y ----------------------------------------------------------------------------Pre-installation Database Requirements - MySQL (concluded) You also must install a MySQL library on the Apache Tomcat instance.Please download the MySQL Connector/J zip file from the link below. Unzip or extract the contents of this file to a directory on your system.The top-level directory should contain a file named: mysql-connector-java-...-bin.jar Copy this file into the /lib directory of your Apache Tomcat 6 installation. OnWindows, the default location for this is 'C:\Program Files\Apache SoftwareFoundation\Tomcat 6.0\lib' Stop and restart your Apache webserver (so it can discover this new library). http://www.mysql.com/downloads/connector/j/5.1.html [Y/n]: y ----------------------------------------------------------------------------Database server settings Specify the host and port number used to communicate with the database server. If the database server is on the same host as the webserver, enter '127.0.0.1' Database port number: [3306]: 3306 Database server hostname: [127.0.0.1]: 127.0.0.1 ----------------------------------------------------------------------------ODK Aggregate database authentication settings The database server must be configured with an account (username and password)that ODK Aggregate will employ for accessing the database server. This is notan account that you will give to anyone; it is only for use by the ODK Aggregatewebserver. Username and password should be alphanumeric strings beginning with aletter (no spaces or punctuation please!). The installer will generate a script that can be run on the database server toestablish the required configuration. Database username: [odk_user]: odk_user Database password: :Retype password: :----------------------------------------------------------------------------ODK Aggregate database datastore settings The database server must be configured with a workspace identified by a databasename, and, if using Postgresql, a schema name. This workspace is where ODKAggregate will store uploaded forms, submissions and other information. Thedatabase name (and schema name for Postgresql) can be any alphanumeric stringsbeginning with a letter; underscores (_) are OK (no spaces or punctuationplease!). The installer will generate a script that can be run on the database server tocreate the workspace. Database Name: [odk_prod]: odk_prod ----------------------------------------------------------------------------ODK Aggregate Instance Name The ODK Aggregate instance name will be displayed to your users when they loginto ODK Aggregate using their ODK Aggregate username and password. Includingthe name of your organization in the instance name can help users confirm thatthey have contacted the correct website. NOTE: During subsequent upgrades, any changes to this text will invalidate allthe ODK Aggregate passwords configured on your webserver. To preventdisruptions during upgrades, please make a record of the name you have chosen. Please enter the name of your instance: []: odk_test ----------------------------------------------------------------------------Super-user ODK Aggregate Username The user with this account will always have full permissions on the system. The first time ODK Aggregate runs, only this user is allowed to log onto thesystem. Upon first logging in, they should change their password and completethe configuration of ODK Aggregate by specifying additional users and whatpermissions those users have on the system. This user's password will be set to'aggregate' upon initial install or upon changing the ODK Aggregate instancename. Please enter an ODK Aggregate Username: []: admin ----------------------------------------------------------------------------Setup is now ready to begin configuring ODK Aggregate. Do you want to continue? [Y/n]: yIf you go to the installation folder (/home/sr/odk) you’ll find three files.

  1. ODKAggregate.war
  2. README.html
  3. create_db_and_user.sql
Login to the MySQL and use the .sql file to create the database.

mysql -u root -psource ~/home/sr/odk/create_db_and_user.sqlYou may want to create a mysql-connector for the Tomcat. Extract the tar file and move the mysql-connector-java-5.1.40.jar file to /opt/tomcat/lib directory.

cd /home/sr/odksudo wget http://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.40.tar.gzsudo tar -xvzf mysql-connector-java-5.1.40.tar.gzcd mysql-connector-java-5.1.40sudo mv mysql-connector-java-5.1.40.jar /opt/tomcat/lib/We are almost there. come back to the ODK installation folder, rename ODKAggregate.war to ROOT.war. Move the ROOT.war file to /opt/tomcat/webapps folder.

sudo mv /home/sr/odk/ODKAggregate.war /opt/tomcat/webapps/ROOT.warIn your browser hit your xxx.xxx.xx.xx:8080/ROOT.war. At first you may end up to a blank page. Hit it again with xxx.xxx.xx.xx:8080. And there is an error.



There is still some things need to be done. Tomcat 8, Glassfish and Jetty require few additional configuration steps to run ODK Aggregate. All of these webservers require configuration settings to enable cookies under HTTPS. Otherwise, ODK Aggregate uses no special Tomcat features and it should operate correctly within any compliant Servlet web container.



Edit the context.xml file of Tomcat.

sudo vi /opt/tomcat/conf/context.xml WEB-INF/web.xml ${catalina.base}/conf/web.xml If you are not familiar with Vim commands, here is a quick tip. Use vi to start edit a file. Hit i to insert or edit a line. Hit escape to exit the insert mode, hit : to enter the menu mode. In menu mode (with : type wq to save edits and exit vi mode. Simply hit q! to exit without saving. We used these commands to add useHttpOnly="false" (line-19) to our file.



Create a directory called WEB-INF inside webbapps folder and create two file here.

cd /opt/tomcat/webapps/sudo mkdir WEB-INFcd WEB-INF/sudo vi glassfish-web.xmlAdd these lines inside the file and save it.

Add another file called jetty-web.xml and add these lines.

sudo vi jetty-web.xml true Remember : to enter menu, wq to save and quit.



Clean the browser cache, restart the browser. Go to xxx.xxx.xx.xx:8080, use admin as username, aggregate as password.







You have successfully reached the shore. Congratz and enjoy!



MORE RESOURCES

Design a form with XLSForm. This is as simple as working with Excel sheets, no coding requires. Here is a sample. Once the form is finished, use this page to convert and upload to ODK Aggregate.



Download and install ODK Collect to you android device. Set your credentials from General Stting>Configure Platform Setting. Once set, check if you can fetch blank form with Get Blank Form. Download the form from there, fill it, save and send finalized form back to Aggregate.



If you don’t have internet with you, try ODK Briefcase. It can pull finalized data from your devise to your PC and send later on. It can even download submitted data from ODK Aggregate in bulk.



You can even publish your data to Fusion table in stream. You can export the data as KML and see on Google Earth.

--------------------------------------------------------------------------------------------------------------------------------



Also published here in ClubGIS.net



أكثر...
 
أعلى