Member-only story
All Software programs work with SQL
All software programs work with SQL.
The only question is: which programming language is used?
I have seen this happen so many times. First, programmers start programming in Python and MongoDB, while experts and developers work with SQL. Then, at some point, someone gets the bright idea to use a tool like Airflow or Kubernetes to run a Hadoop cluster that powers a GraphQL API serving real-time streaming data. But, then they start running into scalability issues because Hadoop and Cassandra are unable to scale to their requirements.
Scaling your GraphQL API
GraphQL API middleware like Apollo, Relay, and gql-http has the nice feature of autoscaling which makes it easy for you to scale your GraphQL API. In almost all cases, you are able to use all the features of GraphQL API middleware including caching, authentication with OIDC, and data fetching. But, schema or data structure for the GraphQL API and the backend database will be the same. This means that the same SQL statements are executed on each request. This leads to the performance hit of SQL queries because Postgres uses specialist databases to handle queries more efficiently than a traditional RDBMS.
Did you know that Kubernetes has a solution for this problem? It is container orchestrator support for stateful applications. This feature allows you to setup an MySQL container for each endpoint on your GraphQL API. It makes network and storage efficient by sending the data to the closest MySQL container.
#dataprocessing #dataanalysis #datapipeline #dataengineering #analyticsengineering
Originally published at https://www.linkedin.com.
