# Data Engineering

- [AWS](https://til.duyet.net/data-engineering/aws.md)
- [Make an NVMe volume available for use on AWS EC2](https://til.duyet.net/data-engineering/aws/make-an-nvme-volume-available-for-use-on-aws-ec2.md)
- [AWS EMR](https://til.duyet.net/data-engineering/aws/emr.md): Manage EMR Cluster
- [S3 Dist CP](https://til.duyet.net/data-engineering/aws/emr/s3-dist-cp.md): Using S3DistCp, you can efficiently copy large amounts of data from Amazon S3/HDFS into S3/HDFS
- [Manage services](https://til.duyet.net/data-engineering/aws/emr/manage-services.md)
- [EMR - Tez](https://til.duyet.net/data-engineering/aws/emr/emr-tez.md): Upscale/Downscale of the emr cluster corrupts hdfs jars /apps/tez/tez.tar.gz
- [Issues](https://til.duyet.net/data-engineering/aws/emr/issues.md)
- [Tools](https://til.duyet.net/data-engineering/tools.md)
- [Shell](https://til.duyet.net/data-engineering/shell.md)
- [MacOS's Touch ID on Terminal](https://til.duyet.net/data-engineering/shell/macoss-touch-id-on-terminal.md)
- [Using \`sed\` to find and replace in file](https://til.duyet.net/data-engineering/shell/using-sed-to-find-and-replace-in-file.md): sed = Stream EDitor
- [Merging contents of multiple .csv files into single .csv file](https://til.duyet.net/data-engineering/shell/merging-contents-of-multiple-.csv-files-into-single-.csv-file.md)
- [env from ConfigMap or Secrets](https://til.duyet.net/data-engineering/env-from-configmap-or-secrets.md)
- [Kubernetes, Helm, Kustomize](https://til.duyet.net/data-engineering/kubernetes.md)
- [initContainer to download file to pod](https://til.duyet.net/data-engineering/kubernetes/initcontainer-to-download-file-to-pod.md)
- [Kustomize: omission of resources](https://til.duyet.net/data-engineering/kubernetes/kustomize-omission-of-resources.md): Skip one of the resources, adding \`$path: delete\` kustomize will skip it.
- [K8S: Services and Labels](https://til.duyet.net/data-engineering/kubernetes/k8s-services-and-labels.md)
- [K8S: PVC stuck in status “Terminating”](https://til.duyet.net/data-engineering/kubernetes/pvc-stuck-in-status-terminating.md)
- [K8S: Port Forward](https://til.duyet.net/data-engineering/kubernetes/k8s-port-forward.md)
- [K8S: Pull an Image from a Private Registry](https://til.duyet.net/data-engineering/kubernetes/pull-an-image-from-a-private-registry.md): Pull Image from Github Docker Package Private
- [Happy helm](https://til.duyet.net/data-engineering/kubernetes/happy-helm.md)
- [Helm: render manifest locally](https://til.duyet.net/data-engineering/kubernetes/helm-render-manifest-locally.md)
- [Helm: Public Helm chart repository with GitHub Pages](https://til.duyet.net/data-engineering/kubernetes/create-public-helm-chart-repository-with-github-pages.md)
- [Apache Airflow](https://til.duyet.net/data-engineering/apache-airflow.md)
- [Extend official Docker image](https://til.duyet.net/data-engineering/apache-airflow/extend-official-docker-image.md)
- [Generate offline SQL upgrade script](https://til.duyet.net/data-engineering/apache-airflow/generate-offline-sql-upgrade-script.md)
- [Airflow in Docker](https://til.duyet.net/data-engineering/apache-airflow/airflow-docker.md)
- [Mastering Airflow UI](https://til.duyet.net/data-engineering/apache-airflow/mastering-airflow-ui.md)
- [Best Practices for Airflow and ETLs](https://til.duyet.net/data-engineering/apache-airflow/best-practices-for-airflow-and-etls.md)
- [Airflow in Docker Compose](https://til.duyet.net/data-engineering/apache-airflow/airflow-in-docker-compose.md): Setup Airflow in Docker Compose
- [Useful SQL queries for Apache Airflow](https://til.duyet.net/data-engineering/apache-airflow/useful-sql-queries-for-apache-airflow.md)
- [Docker](https://til.duyet.net/data-engineering/docker.md)
- ["Distroless" Docker Images](https://til.duyet.net/data-engineering/docker/distroless-docker-images.md)
- [Docker cleanup](https://til.duyet.net/data-engineering/docker/docker-cleanup.md)
- [Optimize the Docker Image Size](https://til.duyet.net/data-engineering/docker/optimize-the-docker-image-size.md)
- [The best Docker base image for Python](https://til.duyet.net/data-engineering/docker/the-best-docker-base-image-for-python.md)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://til.duyet.net/data-engineering.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.