Confirm that the data catalog meets the minimum expectations for infrastructure and security so that it can be maintained long term
The following criteria were to be investigated
- Docker/container setup (via compose, ECS, Fargate etc)
- Role Based Access Control
Docker/container setup (via compose, ECS, Fargate etc)
Deployed a non-production amundsen on AWS ECS using aws-cli.
Amundsen also has some limited documentation on Kubernetes deployment, we have not investigated this.
You can enable OIDC authentication in React application (amundsen frontend).
This was not further investigated in this spike due to setup complications.
- Unable to load the data though the frontend. This is despite the fact that the data was successfully loaded to the Neo4j backend and the data was queryable though elasticsearch
- The Amundsen documentation assumes a level of experience with the tool
- There are no obvious solutions for notifications or alerts
- Role bases access control was not reviewed as part of the spike due to time constraints and deployment issues
- Notifications/alerts was not reviewed as part of the spike due to time constraints and deployment issues
- Investigate kubernetes deployment
- To deploy amundsen, first install the ECS CLI according to the AWS documentation
- Then follow Amundsen documentation
- If you are using aws-vault, you can follow the commands below
Note that you will need to set up a key pair
aws-vault exec hackney-dataplatform-development -- ecs-cli configure --cluster amundsen --region eu-west-2 --default-launch-type EC2 --config-name amundsen
aws-vault exec hackney-dataplatform-development -- ecs-cli configure profile --profile-name amundsen
aws-vault exec hackney-dataplatform-development -- ecs-cli up --keypair test-amundsen --extra-user-data userData.sh --capability-iam --size 1 --instance-type t2.large --cluster-config amundsen --verbose --force
aws-vault exec hackney-dataplatform-development -- ecs-cli compose --cluster-config amundsen --file docker-ecs-amundsen.yml up --create-log-groups
aws-vault exec hackney-dataplatform-development -- ecs-cli ps