Taxonomy and Ontology are the two famous words of the modern AI era powered by everything is connected philosophy - It is a foundational principle of neural networks of our brain and how we store information and retrieve them and widely powers modern storage and retrieval systems. Let it be[...]
Read more
We often pick Embedding models based on public benchmarks like MTEB and various other factors like reputation and popularity etc. but how often we make informed decision by validating the model against our own dataset, on our own machines and curated conditions - Closer to the realtime application/solution To pick[...]
Read more
As you all know I have been working on this AI and Data Space for a while now and I have seen various bottlenecks and complexities in the data engineering and augmentation layer - at both startup and large enterprise clients When you think of where to use AI or[...]
Read more
It all started when I wanted to check the performance of JSON of MYSQL and JSONB of PostgreSQL I am a big fan of PostgreSQL for its power and I have read N number of articles showing why PostgreSQL is better than MySQL Various Benchmarks and research done to prove[...]
Read more
In this article, we will see an interesting use case or system design where we want to connect AWS SNS to an internal HTTP or HTTPS service. SNS is a public service designed to connect to external public HTTPS endpoints. SNS is purpose-built for external communications and it supports Public[...]
Read more
In this article, we are going to see how to find unused load balancers - based on Cloud Watch usage metrics like request count, active flow count We are going to get the usage metric from CloudWatch API using Python Boto and write it as a CSV report for all[...]
Read more
In this article, we are going to see how to configure SendGrid as an SMTP relay for the Metabase If you are here I assume that you already are aware of Metabase and a little basics of SendGrid Mail service provider. In Metabase, the SMTP server or Email Channel plays[...]
Read more
In this article, we are going to see how to save the SNS messages to S3 using Kinesis Data Firehose The first step of building a data engineering pipeline in your organization is storing the unorganized data with long retention and partitioning. Therefore storing the SNS data with Long-term storage[...]
Read more
While using NGINX reverse Proxy as a Sidecar container for your microservice, We often use localhost to connect to the application container on the same POD but it can potentially harm your application availability in certain systems and lead to a lot of 502 or connection refused issues Here is[...]
Read more
Apache Drill is a revolutionary product that identifies itself as a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage - While it was often represented or perceived as a tool for Data Engineers or Analysts. It is not just for Data Engineers In this article, we are going[...]
Read more