Setup ODK-X Sync Endpoint Manually

Follow these setup instructions for manual or local setup of the sync endpoint. If deploying to a cloud service, check out Cloud-based Setup

Prerequisites

You must have Docker 18.09.2 or newer, and be running in Swarm Mode. Follow these links for detailed instructions on installing Docker and enabling Swarm Mode.

If you test single-node swarm, simply run docker swarm init. Use docker info to ensure swarm is activated

If you wish to enable HTTPS, you also need to install certbot

ODK-X Sync Endpoint Setup

ODK-X Sync Endpoint requires a database and a LDAP directory, you could follow the instructions and deploy all three components together or supply your own database and/or LDAP directory.

Note

All of the following commands should be run on your server.

If you are using git on Windows, make sure git is configured with "core.autocrlf=false" - otherwise it will convert line endings with LF to CRLF, which will cause problems with the .sh-files when used in the Docker containers, thus preventing odk/sync-endpoint from starting and instead just returning with an ":invalid argument"-error.

Setup instructions:

  1. Choose a directory to store you endpoint in. In that directory, run:

$ git clone https://github.com/odk-x/sync-endpoint-default-setup

  1. Navigate into the the "sync-endpoint-default-setup" directory

  2. Checkout the sync-endpoint code by running:

$ git clone -b master --single-branch --depth=1 https://github.com/odk-x/sync-endpoint

Note

The above command performs a shallow clone of the sync-endpoint GitHub git repository. To perform a full clone remove –single-branch –depth=1 from the command.

  1. Navigate into the sync-endpoint directory. Most likely

$ cd sync-endpoint

  1. Build sync endpoint by running the following: (NOTE: you will need Apache Maven installed >= 3.3.3)

$ mvn clean install

  1. Navigate back to the parent "sync-endpoint-default-setup" directory.

  2. In the "sync-endpoint-default-setup" directory run:

$ docker build --pull -t odk/sync-web-ui https://github.com/odk-x/sync-endpoint-web-ui.git

  1. In the "sync-endpoint-default-setup" cloned repository run:

$ docker build --pull -t odk/db-bootstrap db-bootstrap

  1. In the "sync-endpoint-default-setup" cloned repository run:

$ docker build --pull -t odk/openldap openldap

  1. In the "sync-endpoint-default-setup" cloned repository run:

$ docker build --pull -t odk/phpldapadmin phpldapadmin

  1. Enter your hostname in the security.server.hostname field (if such field doesn't exists, create one at the bottom of file) in the security.properties file (under the directory config/sync-endpoint). You can also choose to enable Anonymous access on your ODK-X Sync Endpoint by configuring the same security.properties file.

  2. If you're not using the standard ports (80 for HTTP and 443 for HTTPS) enter the ports you're using in the security.server.port and security.server.securePort fields in the security.properties (if such a field doesn't exists, create it at the bottom of file). Then add or edit the ports section under the sync section in docker-compose.yml to be YOUR_PORT:8080.

Note

It is important that the right side of the colon stays as 8080 or whatever port you are using (8080 is the default). This is the port that the web server is looking for. Any other services running on port:8080 need to be stopped as it will prevent the server from running, for example: Apache2

Reminder that only one process can own a port at a time so if another process on the computer is using port 8080 there will be a conflict and sync-endpoint may not function correctly.

  1. If you're using your own LDAP directory or database, continue with the instructions:

  1. In the "sync-endpoint-default-setup" cloned repository run:

  • For HTTP:

$ docker stack deploy -c docker-compose.yml syncldap

  • For HTTPS:

$ docker stack deploy -c docker-compose.yml -c docker-compose-https.yml syncldap

If there is a failure during the docker stack deploy process, try take the docker stack down first and bring it back up again with the previous same docker stack deploy command.

  1. The server takes about 30s to start, then it will be running at http://127.0.0.1.

  2. See the LDAP section for instructions on configuring users and groups.

  3. See the Stop the ODK-X Sync Endpoint section to stop the service.

Stopping ODK-X Sync Endpoint

  1. Run:

$ docker stack rm syncldap

  1. OPTIONAL: If you want to remove the volumes as well,

Warning

Removing volumes will remove any provisioned TLS keys if https is enabled. These keys can only be provisioned at a rate of 50 valid keys/domain/week.

  • Linux/macOS:

$ docker volume rm $(docker volume ls -f "label=com.docker.stack.namespace=syncldap" -q)

  • Windows:

$ docker volume rm (docker volume ls -f "label=com.docker.stack.namespace=syncldap" -q)

Custom database

  1. If you haven't followed the common instructions, start with those.

  2. Remove the db and db-bootstrap sections in docker-compose.yml.

  3. Modify jdbc.properties`(under the directory :file:`config/sync-endpoint) to match your database. Supported database systems are PostgreSQL, MySQL and Microsoft SQL Server. You can find the minimum tested versions of MySQL, PostgreSQL, and MSSQL that are compatible with Sync-Endpoint in the GitHub repository. Sample config for PostgreSQL can be found on Github, and below are some more detailed config for each type of database.

    • jdbc.driverClassName=

      • org.postgresql.Driver (PostgreSQL)

      • com.mysql.jdbc.Driver (MySQL)

      • com.microsoft.sqlserver.jdbc.SQLServerDriver (Microsoft SQL Server)

    • jdbc.resourceName=jdbc/YOUR_DATASOURCE

    • jdbc.url=

      • jdbc:postgresql://YOUR_SERVER/YOUR_DATABASE?param1=value1&param2=value2&... (PostgreSQL)

      • jdbc:mysql://YOUR_SERVER/YOUR_DATABASE?param1=value1&param2=value2&... (MySQL)

      • jdbc:sqlserver://YOUR_SERVER;database=YOUR_DATABASE;param1=value1;param2=value2;... (Microsoft SQL Server)

    • jdbc.username=YOUR_USERNAME

    • jdbc.password=YOUR_PASSWORD

    • jdbc.schema=YOUR_SCHEMA

  4. Modify sync.env to match your database

  5. In the cloned repository,

$ docker stack deploy -c docker-compose.yml syncldap

  1. The server takes about 30s to start, then it will be running at http://127.0.0.1.

Custom LDAP directory

  1. If you haven't followed the common instructions, start with those.

  2. OPTIONAL: If your LDAP directory uses a certificate that was signed by a self-signed CA,

  1. Make the public key of the CA available to ODK-X Sync Endpoint with this command.

$ docker config create org.opendatakit.sync.ldapcert PATH_TO_CERT

  1. Uncomment the relevant lines in the configs section in docker-compose.yml and the configs section under the sync section in docker-compose.yml.

  1. Create a new directory in the sync-endpoint-default-setup directory and create a Docker file inside it.

  2. Copy the bootstrap.ldif file from the OpenLDAP directory to the new directory. In the Docker file Add the image of the LDAP Directory to be used and add the "COPY" command to copy the bootstrap.ldif file to the right path in the container.

  3. Run the following command to build the Docker image :

$ docker build -t odk/[LDAP_DIRECTORY_NAME] [ Folder conatining the Docker file ]

  1. Replace the ldap-service image from docker-compose.yml with odk/[LDAP_DIRECTORY_NAME].

  2. In the sync-endpoint-default-setup directory navigate to config/sync-endpoint. Modify the security.properties file to fill in the Settings for LDAP server. Set security.server.ldapUrl in security.properties to the new server url. The name of the service in Swarm would be same ( ldap-service ). So just change the port number. After this following settings need to be configured in the same file for the LDAP server:

    • security.server.ldapBaseDn

    • security.server.ldapPooled

    • security.server.userSearchBase

    • security.server.groupSearchBase

    • security.server.groupRoleAttribute

    • security.server.userFullnameAttribute

    • security.server.usernameAttribute

    • security.server.userDnPattern

    • security.server.memberOfGroupSearchFilter

    • security.server.serverGroupSearchFilter

Note

The LDAP Directory here is configured to run inside the Docker Swarm. If you are running the LDAP Directory outside the Docker Swarm and it is accessible for the containers inside the Docker Swarm, you can directly follow step 7 to configure it.

Note

The default configuration does not use ldaps or StartTLS because the LDAP directory communicates with the ODK-X Sync Endpoint over a secure overlay network. You should use ldaps or StartTLS to communicate with your LDAP directory.

  1. In the cloned repository:

$ docker stack deploy -c docker-compose.yml syncldap

  1. The server takes about 30s to start, then it will be running at http://127.0.0.1.

Anonymous Access for ODK-X Sync Endpoint

Checking for Anonymous User Access

If you have already created the Docker Config and deployed the Docker Stack. Navigate to http://[IP_ADDRESS]/web-ui/admin/users or http://[IP_ADDRESS]/odktables/[APP_NAME]/usersInfo

Users and Permissions

User ID

Full Name

Membership Roles

anonymous

Anonymous Access

ROLE_USER, ROLE_SYNCHRONIZE_TABLES

If you find a user with attributes as shown above then your server has Anonymous User Access. If not then you can easily add Anonymous User Access by following Enabling or Disabling Anonymous User Access.

Enabling or Disabling Anonymous User Access

  1. If you have deployed the Docker Stack then may want to Stop the ODK-X Sync Endpoint Server before proceeding.

  2. Navigate to security.properties file which can be found under sync-endpoint-default-setup/config/sync-endpoint/ directory.

  • To Enable Anonymous access set the following fields to true

    sync.preference.anonymousTablesSync=true
sync.preference.anonymousAttachmentAccess=true

  • To Disable Anonymous access set the following fields to false

    sync.preference.anonymousTablesSync=false
sync.preference.anonymousAttachmentAccess=false

3. Update the Docker Config by either recreating it or redeploying the Docker Stack. You can redeploy the stack using the following command.

$ docker stack deploy -c /root/sync-endpoint-default-setup/docker-compose.yml syncldap