[{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/26.04.1/","section":"Tags","summary":"","title":"26.04.1"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/","section":"Categories","summary":"","title":"Categories"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/","section":"dbaStreet Blog","summary":"","title":"dbaStreet Blog"},{"content":"I have been using Ubuntu on my personal desktops and laptops exclusively since 2008. I like using ubuntu for its stability, easy of installation and use. So it is no surprise that i am running Ubuntu on my hp 255 G10 laptop. I had ubuntu 25.10 installed and it ran out of support, so i decided to install the latest version of ubuntu which is an LTS version 26.04. As years of experience has taught me, i never install the firt release of a major version of anything, may it be oracle , postgres or ubuntu Lol. I wait for the first point release after the major release, so that most of the initial bugs are ironed out. Ubuntu releaed 26.04.1 on Aug 27 2026.\nI could have upgraded my laptop from 25.10 to 26.04 but every 6 months or so i like to do a fresh install. As i work on my laptop i end up adding a lot of packages and making a lot of configuration changes which over time, results in a lot of stuff i rarely use on the laptop. Lot of junk and stuff gets slower. So i like starting fresh. This was such an opportunity. So i made backups of all the files i needed and did a fresh install of Ubuntu 26.04.1\nI installed the startup disk creator app on ubuntu, downloaded the Ubuntu iso for x86-64, and used the startup disk creator to create a bootable image on my usb. Restarted the laptop, press \u0026ldquo;ESC\u0026rdquo; a few times to get the boot menu and choose to boot from the USB drive and follow the prompts to completely erase my previous install and do a fresh install.\nOnce the install was completed, i installed uv, python, rust, nodejs,podman,psql and hugo,restored a few files i needed from my backup, and I was off to the races. Loving Ubuntu 26.04.1 (my umpteenth ubuntu distribution) sofar.\n","date":"5 September 2026","permalink":"https://dbastreetblog.b-cdn.net/posts/post2026-07-ubuntuinstall/","section":"Posts","summary":"","title":"Installing Ubuntu 26.04.1"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/linux/","section":"Categories","summary":"","title":"Linux"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/linux/","section":"Tags","summary":"","title":"Linux"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/ubuntu/","section":"Categories","summary":"","title":"Ubuntu"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/ubuntu/","section":"Tags","summary":"","title":"Ubuntu"},{"content":"In this post I will discuss the steps to create a logical replication publication from postgresql and how to read from the replication slot using a rust program. We will be using the native postgres default decoder, pgoutput in these steps. When you read from the replication slot you get the data in a binary protocol (So you dont see actual insert/update/delete statements).\nPre-requisites # In the postgres database set the configuration parameter wal_level=\u0026lsquo;logical\u0026rsquo;,max_wal_senders=10 and max_replication_slots=10 (10 or a reasonable number) Postgresql steps #From the psql client connect to the postgresql database and do the following\ncreate table books (bookid serial,bookname varchar(100),bookauthor varchar(50),bookprice numeric(5,2)); insert into books SELECT floor(30000 * random()),\u0026#39;Author 1\u0026#39;,ceil(random() * 50 + 1) FROM generate_series(1, 10); alter table books replica identity full; SELECT pg_create_logical_replication_slot(\u0026#39;rkrepslot1\u0026#39;, \u0026#39;pgoutput\u0026#39;); CREATE PUBLICATION rklrpub1 FOR TABLE books; This creates a table books, enables replica identity full on the table ,creates a replication slot and creates a publication.\nRust steps #This assumes that you have a rust environment setup and cargo available to create and compile the rust programs.\ncargo new pg-lr cd pg-lr cargo add pgwire_replication This creates a new rust project and adds the crate pgwire_replication to the project. We will use pwire_replication to read from the the postgres replication slot.\nAdd the following code to src/main.rs\nuse pgwire_replication::{ReplicationClient, ReplicationConfig, ReplicationEvent, Lsn}; use std::error::Error; #[tokio::main] async fn main() -\u0026gt; Result\u0026lt;(), Box\u0026lt;dyn Error\u0026gt;\u0026gt; { // FIX: Build it with new() and chain the configuration modifiers let config = ReplicationConfig::new( \u0026#34;localhost\u0026#34;, // host \u0026#34;pg1user\u0026#34;, // user \u0026#34;pg1pwd\u0026#34;, // password \u0026#34;pg1db\u0026#34;, // database \u0026#34;rkrepslot1\u0026#34;, // logical replication slot \u0026#34;rklrpub1\u0026#34;, // Postgres publication name ) .with_port(5432) .with_start_lsn(Lsn::ZERO); println!(\u0026#34;Connecting to PostgreSQL logical replication stream...\u0026#34;); let mut client = ReplicationClient::connect(config).await?; println!(\u0026#34;Successfully connected. Awaiting database records...\u0026#34;); while let Some(event) = client.recv().await? { match event { ReplicationEvent::XLogData { wal_end, data, .. } =\u0026gt; { println!(\u0026#34;--- New WAL Record Received ---\u0026#34;); println!(\u0026#34;WAL End LSN: {}\u0026#34;, wal_end); println!(\u0026#34;Raw Data Size: {} bytes\u0026#34;, data.len()); println!(\u0026#34;Raw Bytes (Lossy String): {:?}\u0026#34;, String::from_utf8_lossy(\u0026amp;data)); client.update_applied_lsn(wal_end); } _ =\u0026gt; {} } } Ok(()) } Modify the hostname, username, password, database name,replication slot name and publication name in the code above to be appropriate for your environment.\ncargo run As you perform dml on the postgresql database the rust program will read from the replication slot and display the binary stream like shown below.\nConnecting to PostgreSQL logical replication stream... Successfully connected. Awaiting database records... --- New WAL Record Received --- WAL End LSN: 0/0 Raw Data Size: 94 bytes Raw Bytes (Lossy String): \u0026#34;R\\0\\0@\\u{2}public\\0books\\0f\\0\\u{4}\\u{1}bookid\\0\\0\\0\\0\\u{17}����\\u{1}bookname\\0\\0\\0\\u{4}\\u{13}\\0\\0\\0h\\u{1}bookauthor\\0\\0\\0\\u{4}\\u{13}\\0\\0\\06\\u{1}bookprice\\0\\0\\0\\u{6}�\\0\\u{5}\\0\\u{6}\u0026#34; --- New WAL Record Received --- WAL End LSN: 0/1C185B0 Raw Data Size: 51 bytes Raw Bytes (Lossy String): \u0026#34;I\\0\\0@\\u{2}N\\0\\u{4}t\\0\\0\\0\\u{5}22437t\\0\\0\\0\\u{b}Second Bookt\\0\\0\\0\\u{2}42t\\0\\0\\0\\u{5}22.00\u0026#34; --- New WAL Record Received --- WAL End LSN: 0/1C18A60 Raw Data Size: 97 bytes Raw Bytes (Lossy String): \u0026#34;U\\0\\0@\\u{2}O\\0\\u{4}t\\0\\0\\0\\u{5}22437t\\0\\0\\0\\u{b}Second Bookt\\0\\0\\0\\u{2}42t\\0\\0\\0\\u{5}22.00N\\0\\u{4}t\\0\\0\\0\\u{5}22437t\\0\\0\\0\\u{b}Second Bookt\\0\\0\\0\\u{2}42t\\0\\0\\0\\u{5}22.50\u0026#34; ","date":"19 July 2026","permalink":"https://dbastreetblog.b-cdn.net/posts/post2026-06-pgrustlr/","section":"Posts","summary":"","title":"Consuming Postgres Native logical replication stream from rust"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/logical-replication/","section":"Tags","summary":"","title":"Logical Replication"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/pgwire_replication/","section":"Tags","summary":"","title":"Pgwire_replication"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/postgresql/","section":"Categories","summary":"","title":"Postgresql"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/postgresql/","section":"Tags","summary":"","title":"Postgresql"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/rust/","section":"Categories","summary":"","title":"Rust"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/rust/","section":"Tags","summary":"","title":"Rust"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/container/","section":"Tags","summary":"","title":"Container"},{"content":"In my previous post i outlined how to create a container with the latest postgresql version (At the time of writing postgres 18) running with a few extensions installed. podman is a daemonless, open source, linux native tool designed to make it easy to find, run, build, share and deploy applications using Open Containers Initiative (OCI) Containers. Using podman avoids the additional overhead of needing to have the docker deamon running and is simpler to use.\nPostgresql Official images for Docker\nThe steps outlined below, composes a new local container image from the latest postgres image available in the docker hub, and runs a container based on the new image.\nInstall the podman binary for your linux platform.\nFirst I create a directory to hold my podman compose files. All of the following steps are run as the non-root user we installed podman with.\nmkdir ~/Apps/docker/pg/pg1 cd ~/Apps/docker/pg/pg1 Create a file Named Containerfile and add the following lines to it. Here we are specifying that podman use the image tagged postgres:latest (This will be the latest generally available version of postgres which is 18 at the time of writing), and install the packages postgresql-contrib (Gets you all the extensions that are contrib modules) and pgvector.\nFROM docker.io/library/postgres:latest # Update package lists RUN apt-get update # Install PostgreSQL extensions RUN apt-get install -y --no-install-recommends \\ postgresql-contrib \\ postgresql-18-pgvector # Clean up RUN rm -rf /var/lib/apt/lists/* Run the following command to build the new image from the base image postgres:latest. This simply builds a new image and does not create a container or start it.\npodman build . Run the following command to list the image we just created and note down the imageid (We will use it in the next step to run the container using this image). You will see 4 images being listed, pick the image that is the latest (use the created column to determine the latest image) (podman creates some intermediate images, as it is building the image by adding the extensions).\npodman image ls -a Run the following command to run a container based on the image we just built. Replace the username, password and database name in the command below with your choices. Replace 27e3aca52506 with the image id we gathered up from the previous step.\npodman run -d -p 5432:5432 --name pg1local -e POSTGRES_PASSWORD=pg1pwd -e POSTGRES_USER=pg1user -e POSTGRES_DB=pg1db \\ -v /home/rkadmin/Apps/docker/pg/pg2/postgresql.conf:/etc/postgresql/postgresql.conf 27e3aca52506 -c \u0026#39;config_file=/etc/postgresql/postgresql.conf\u0026#39; In the above command you see that I am using the -v switch to map a file on my local linux host /home/rkadmin/Apps/docker/pg/pg2/postgresql.conf to the file /etc/postgresql/postgresql.conf on the container. I have created a local file postgresql.conf where i have made the necessary configurations to the postgresql parameters that i want for my postgres database (eg: setting wal_level=\u0026lsquo;logical\u0026rsquo; to enable logical replication). The part of the command -c \u0026lsquo;config_file=/etc/postgresql/postgresql.conf\u0026rsquo; tells the container to use the file as the postgres configuration file (because the file is in the non default location).\nNow you can go ahead and access the database using psql. Once you login create the extensions.\npsql -h localhost -d pg1db -U pg1user psql\u0026gt; create extension pg_stat_statements; psql\u0026gt; create extension vector; ","date":"19 July 2026","permalink":"https://dbastreetblog.b-cdn.net/posts/post2026-05-podman-pg-1/","section":"Posts","summary":"","title":"Creating a postgres container using podman with Extensions"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/extensions/","section":"Tags","summary":"","title":"Extensions"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/podman/","section":"Categories","summary":"","title":"Podman"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/podman/","section":"Tags","summary":"","title":"Podman"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/postgres/","section":"Tags","summary":"","title":"Postgres"},{"content":"In my previous post i outlined how to run Docker in rootless mode on Ubuntu. Once we have that out of the way and we have Docker installed, in this post I\u0026rsquo;ll show you how to create a container with the latest postgresql version (At the time of writing postgres 18) running with a few extensions installed.\nPostgresql Official images for Docker\nThe steps outlined below, composes a new local docker image from the latest postgres image available in the docker hub, and runs a container based on the new image.\nFirst I create a directory to hold my Docker compose files. All of the following steps are run as the non-root user we installed Docker with.\nmkdir ~/Apps/docker/pg/pg1 cd ~/Apps/docker/pg/pg1 Create a file Named Dockerfile and add the following lines to it. Here we are specifying that docker use the image tagged postgres:latest (This will be the latest generally available version of postgres which is 18 at the time of writing), and install the packages postgresql-contrib (Gets you all the extensions that are contrib modules) and pgvector.\nFROM postgres:latest # Update package lists RUN apt-get update # Install PostgreSQL extensions RUN apt-get install -y --no-install-recommends \\ postgresql-contrib \\ postgresql-18-pgvector Create a file named docker-compose.yml and add the following lines to it.Here we give the container a name postgres_db and specify postgres startup parameter that set the extension names that will be preloaded upon postgres startup.\nservices: postgres: container_name: postgres_db # Build a custom image from the local Dockerfile build: . ports: - \u0026#34;5432:5432\u0026#34; # Preload required libraries at startup (comma-separated if multiple) command: \u0026gt; postgres -c shared_preload_libraries=\u0026#39;pg_stat_statements,vector\u0026#39; Run the following command to compose the new image from the base image postgres:latest. This simply builds a new image and does not create a container or start it.\ndocker compose build Run the following command to list the image we just created and note down the imageid (We will use it in the next step to run the container using this image)\ndocker image ls -a Run the following command to run a container based on the image we just built. Replace the username, password and database name in the command below with your choices. Replace 27e3aca52506 with the image id we gathered up from the previous step.\ndocker run -d -p 5432:5432 --name pg1local -e POSTGRES_PASSWORD=pg1pwd -e POSTGRES_USER=pg1user -e POSTGRES_DB=pg1db 27e3aca52506 -c shared_preload_libraries=\u0026#39;pg_stat_statements,vector\u0026#39; Now you can go ahead and access the database using psql. Once you login create the extensions.\npsql -h localhost -d pg1db -U pg1user psql\u0026gt; create extension pg_stat_statements; psql\u0026gt; create extension vector; ","date":"16 June 2026","permalink":"https://dbastreetblog.b-cdn.net/posts/post2026-04-docker-pg-2/","section":"Posts","summary":"","title":"Creating a postgres Docker container with Extensions"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/docker/","section":"Categories","summary":"","title":"Docker"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/docker/","section":"Tags","summary":"","title":"Docker"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/install/","section":"Tags","summary":"","title":"Install"},{"content":"I find it very convenient to run docker containers on my Ubuntu Linux Laptop, so that i can quickly crank up docker images with the latest database versions of postgres, and do testing. When you install Docker on ubuntu, by default, the docker deamon dockerd runs as root.\nI prefer to run the docker deamon process as a non root ubuntu user, below are the steps that i follow in order to set this up.\nDocker Installation Instructions - Ubuntu\nFirst I configure the apt repositories for Ubuntu, running the following commands as root\n# Add Docker\u0026#39;s official GPG key: sudo apt update sudo apt install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the repository to Apt sources: sudo tee /etc/apt/sources.list.d/docker.sources \u0026lt;\u0026lt;EOF Types: deb URIs: https://download.docker.com/linux/ubuntu Suites: $(. /etc/os-release \u0026amp;\u0026amp; echo \u0026#34;${UBUNTU_CODENAME:-$VERSION_CODENAME}\u0026#34;) Components: stable Architectures: $(dpkg --print-architecture) Signed-By: /etc/apt/keyrings/docker.asc EOF sudo apt update Then I install the Docker packages, again as root\nsudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin Verify that Docker is running\nsudo systemctl status docker Now we can configure ubuntu to run as a non root user. Docker - Rootless Configuration. First we stop the docker deamon running as root and disable it for good.\nsudo systemctl disable --now docker.service docker.socket sudo rm /var/run/docker.sock Now switch to the Ubuntu user (Non-root) where you want the Docker engine to be running from.\ndockerd-rootless-setuptool.sh install Run the following command to confirm that the docker client is connecting to the rootless deamon\ndocker info Now you have Docker running as a non-root user on Ubuntu\n","date":"16 June 2026","permalink":"https://dbastreetblog.b-cdn.net/posts/post2026-03-docker-pg-1/","section":"Posts","summary":"","title":"Installing Docker rootless mode on Ubuntu"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/rootless/","section":"Tags","summary":"","title":"Rootless"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/chroma/","section":"Categories","summary":"","title":"Chroma"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/chroma/","section":"Tags","summary":"","title":"Chroma"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/embeddings/","section":"Tags","summary":"","title":"Embeddings"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/langchain/","section":"Tags","summary":"","title":"Langchain"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/python/","section":"Categories","summary":"","title":"Python"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/python/","section":"Tags","summary":"","title":"Python"},{"content":"Chroma calls itself the data infrastructure for AI. It allows you to store vector embeddings, text, images and associated metadata and allows you to search through them efficiently. A very popular use case for Chroma would be for RAG (Retrieval augmented generation) in GenAI workflows. You may have pdf documents that contain information (eg: stock analysis reports, SEC filings, medical journal articles) that you can create vector embeddings for and store them in Chroma. Then you can write a script where you can ask a question and you can do a vector similarity search using the question string against the vector embeddings stored in Chroma and the results can be fed to your LLM to summarize. If you use the disk based storage option for Chroma , it will store the metadata related to the document in a sqlite3 database and it stores the actual vector embeddings in files it creates on disk. You can directly feed text documents to Chroma and it can create the embeddings for you too.\nBelow is a script you can use to provide a pdf file as the input, vectorize it and store it in Chroma and create a langchain process to answer a question you ask based on that data. This script uses Amazon Titan for creating the embeddings and uses Amazon Nova lite for summarizing and answering your question.\nimport os from langchain_community.document_loaders import PyPDFLoader from langchain_text_splitters import RecursiveCharacterTextSplitter from langchain_aws import BedrockEmbeddings from langchain_chroma import Chroma from langchain_aws import ChatBedrock from langchain_core.prompts import ChatPromptTemplate from langchain_core.output_parsers import StrOutputParser from langchain_core.runnables import RunnablePassthrough import argparse parser = argparse.ArgumentParser() parser.add_argument(\u0026#39;-f\u0026#39;, \u0026#39;--file\u0026#39;, help=\u0026#34;File Name\u0026#34;, required=True) parser.add_argument(\u0026#39;-m\u0026#39;, \u0026#39;--model\u0026#39;, choices=[\u0026#39;us.anthropic.claude-3-7-sonnet-20250219-v1:0\u0026#39;,\u0026#39;us.anthropic.claude-3-5-sonnet-20241022-v2:0\u0026#39;,\u0026#39;anthropic.claude-3-5-sonnet-20240620-v1:0\u0026#39;,\u0026#39;us.amazon.nova-lite-v1:0\u0026#39;],help=\u0026#34;Model Name choose one of us.anthropic.claude-3-7-sonnet-20250219-v1:0,us.anthropic.claude-3-5-sonnet-20241022-v2:0,anthropic.claude-3-5-sonnet-20240620-v1:0,us.amazon.nova-lite-v1:0\u0026#34;, required=False) args = parser.parse_args() if not os.path.isfile(args.file): print(\u0026#39;Cannot access \u0026#39;+args.file) quit() if args.model is None: l_fm=\u0026#39;us.amazon.nova-lite-v1:0\u0026#39; elif args.model: l_fm=args.model else: print(\u0026#39;Invalid Model \u0026#39;+args.model) quit() print(\u0026#39;Using FM \u0026#39;+l_fm) # Initialize Bedrock client def init_bedrock_client(): bedrock_client = ChatBedrock( region_name=\u0026#34;us-east-1\u0026#34;, model_id=l_fm, model_kwargs=dict(temperature=0.7) ) return bedrock_client # Load and chunk the PDF def process_pdf(file_path): # Load PDF loader = PyPDFLoader(file_path) documents = loader.load() # Create text splitter text_splitter = RecursiveCharacterTextSplitter( chunk_size=1000, chunk_overlap=100, separators=[\u0026#34;\\n\\n\u0026#34;, \u0026#34;\\n\u0026#34;, \u0026#34;.\u0026#34;, \u0026#34;!\u0026#34;, \u0026#34;?\u0026#34;, \u0026#34;,\u0026#34;, \u0026#34; \u0026#34;, \u0026#34;\u0026#34;], length_function=len ) # Split documents into chunks chunks = text_splitter.split_documents(documents) return chunks # Create vector store def create_vector_store(chunks): # Initialize Bedrock embeddings embeddings = BedrockEmbeddings(region_name=\u0026#34;us-east-1\u0026#34;,model_id=\u0026#34;amazon.titan-embed-text-v2:0\u0026#34;) # Create vector store vector_store = Chroma.from_documents(chunks, embeddings,collection_name=\u0026#34;hcls_collection\u0026#34;,persist_directory=\u0026#34;./chroma_db\u0026#34;) return vector_store # Setup QA chain def setup_qa_chain(vector_store): llm = init_bedrock_client() retriever = vector_store.as_retriever(search_kwargs={\u0026#34;k\u0026#34;: 3}) prompt_template = ChatPromptTemplate.from_template( \u0026#34;Answer the question based only on the following context:\\n\\n\u0026#34; \u0026#34;{context}\\n\\n\u0026#34; \u0026#34;Question: {user_input}\u0026#34; ) output_parser = StrOutputParser() qa_chain = ( {\u0026#34;context\u0026#34;: retriever, \u0026#34;user_input\u0026#34;: RunnablePassthrough()} | prompt_template | llm | output_parser ) return qa_chain def main(): # Path to your PDF file pdf_path = args.file # Process the PDF print(\u0026#34;Processing PDF...\u0026#34;) chunks = process_pdf(pdf_path) print(f\u0026#34;Created {len(chunks)} chunks\u0026#34;) # Create vector store print(\u0026#34;Creating vector store...\u0026#34;) vector_store = create_vector_store(chunks) # Setup QA chain print(\u0026#34;Setting up QA chain...\u0026#34;) qa_chain = setup_qa_chain(vector_store) # Interactive question answering while True: question = input(\u0026#34;\\nEnter your question (or \u0026#39;quit\u0026#39; to exit): \u0026#34;) if question.lower() == \u0026#39;quit\u0026#39;: break try: response = qa_chain.invoke(question) print(\u0026#34;\\nAnswer:\u0026#34;, {response}) except Exception as e: print(f\u0026#34;Error: {str(e)}\u0026#34;) if __name__ == \u0026#34;__main__\u0026#34;: main() You will run the script by passing the pdf file you want to store in Chroma as shown below\npython3 python-chroma.py -f your-file-name-here.pdf The first time you run the script it will create a subdirectory ./chroma-db under which it will store the chroma related files. If you have sqlite3 installed on your machine you can open up the sqlite database (There will be a .sqlite file in the ./chroma-db directory) and check the table names and its contents.\n","date":"31 March 2026","permalink":"https://dbastreetblog.b-cdn.net/posts/post2026-02-py-chroma-1/","section":"Posts","summary":"","title":"Using ChromaDB as a vector store with python"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/vector-index/","section":"Tags","summary":"","title":"Vector Index"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/vector-store/","section":"Tags","summary":"","title":"Vector Store"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/aurora/","section":"Categories","summary":"","title":"Aurora"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/cloudwatch/","section":"Tags","summary":"","title":"Cloudwatch"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/metrics/","section":"Tags","summary":"","title":"Metrics"},{"content":"When you are using Aurora Serverless V2 , you can get resource utilization metrics from 3 sources, basic cloudwatch metrics, enhanced monitoring metrics and cloudwatch database insights. This python script below queries metrics like CPUUtilization, ACUUtilization, ServerlessDatabaseCapacity at the instance level and shows them side by side in a table.\nThe script takes the instance id and begin and end times as arguments and displays the following metrics\nCloudwatch Metrics #CPUUtilization,FreeableMemory,ConnectionAttempts,BufferCacheHitRatio,ReadIOPS,ReadLatency,ReadThroughput\nPerformance insights Metrics #CPUUtilization,os.cpuUtilization.total.avg,os.tasks.running.avg,os.general.serverlessDatabaseCapacity.avg,os.general.maxConfiguredAcu.avg,os.general.numVCPUs.avg,db.User.numbackends.avg,db.SQL.queries_started.avg,os.diskIO.auroraStorage.diskQueueDepth.avg,os.memory.free.avg}\nimport boto3 import json import argparse from datetime import datetime, timedelta,timezone,UTC import pandas as pd parser = argparse.ArgumentParser() parser.add_argument(\u0026#39;-i\u0026#39;, \u0026#39;--inst\u0026#39;, help=\u0026#34;Instance Id\u0026#34;, type=str,required=True) parser.add_argument(\u0026#39;-b\u0026#39;, \u0026#39;--begintime\u0026#39;, help=\u0026#34;Begin Time\u0026#34;,type=str,required=False,default=\u0026#39;26/03/18 00:01\u0026#39;) parser.add_argument(\u0026#39;-e\u0026#39;, \u0026#39;--endtime\u0026#39;, help=\u0026#34;End Time\u0026#34;,type=str,required=False,default=\u0026#39;26/03/18 23:59\u0026#39;) args = parser.parse_args() pd.set_option(\u0026#39;display.float_format\u0026#39;, lambda x: \u0026#39;%.2f\u0026#39; % x) l_btime=datetime.strptime(args.begintime,\u0026#34;%y/%m/%d %H:%M\u0026#34;) l_etime=datetime.strptime(args.endtime,\u0026#34;%y/%m/%d %H:%M\u0026#34;) instance_id = args.inst rds = boto3.client(\u0026#34;rds\u0026#34;) desc = rds.describe_db_instances(DBInstanceIdentifier=instance_id) dbi_resource_id = desc[\u0026#34;DBInstances\u0026#34;][0][\u0026#34;DbiResourceId\u0026#34;] # now = datetime.now(UTC) # start = now - timedelta(hours=1) now = l_etime start = l_btime start_cwlog = start - timedelta(hours=4) now_cwlog = now - timedelta(hours=4) # 1. Cloudwatch metrics df_cw_metrics=pd.DataFrame(); d_cw_col_names={\u0026#34;CPUUtilization\u0026#34;:\u0026#34;oscpuutil\u0026#34;,\u0026#34;FreeableMemory\u0026#34;:\u0026#34;freabmem\u0026#34;,\u0026#34;ConnectionAttempts\u0026#34;:\u0026#34;conatt\u0026#34; ,\u0026#34;BufferCacheHitRatio\u0026#34;:\u0026#34;bchitrat\u0026#34;,\u0026#34;ReadIOPS\u0026#34;:\u0026#34;riops\u0026#34; ,\u0026#34;ReadLatency\u0026#34;:\u0026#34;rlat\u0026#34;,\u0026#34;ReadThroughput\u0026#34;:\u0026#34;rthr\u0026#34;} cw = boto3.client(\u0026#39;cloudwatch\u0026#39;) metrics = [ {\u0026#39;Namespace\u0026#39;: \u0026#39;AWS/RDS\u0026#39;, \u0026#39;MetricName\u0026#39;: \u0026#39;CPUUtilization\u0026#39;, \u0026#39;Dimensions\u0026#39;: [{\u0026#39;Name\u0026#39;: \u0026#39;DBInstanceIdentifier\u0026#39;, \u0026#39;Value\u0026#39;: instance_id}]}, {\u0026#39;Namespace\u0026#39;: \u0026#39;AWS/RDS\u0026#39;, \u0026#39;MetricName\u0026#39;: \u0026#39;FreeableMemory\u0026#39;, \u0026#39;Dimensions\u0026#39;: [{\u0026#39;Name\u0026#39;: \u0026#39;DBInstanceIdentifier\u0026#39;, \u0026#39;Value\u0026#39;: instance_id}]}, {\u0026#39;Namespace\u0026#39;: \u0026#39;AWS/RDS\u0026#39;, \u0026#39;MetricName\u0026#39;: \u0026#39;BufferCacheHitRatio\u0026#39;, \u0026#39;Dimensions\u0026#39;: [{\u0026#39;Name\u0026#39;: \u0026#39;DBInstanceIdentifier\u0026#39;, \u0026#39;Value\u0026#39;: instance_id}]}, {\u0026#39;Namespace\u0026#39;: \u0026#39;AWS/RDS\u0026#39;, \u0026#39;MetricName\u0026#39;: \u0026#39;ReadIOPS\u0026#39;, \u0026#39;Dimensions\u0026#39;: [{\u0026#39;Name\u0026#39;: \u0026#39;DBInstanceIdentifier\u0026#39;, \u0026#39;Value\u0026#39;: instance_id}]}, {\u0026#39;Namespace\u0026#39;: \u0026#39;AWS/RDS\u0026#39;, \u0026#39;MetricName\u0026#39;: \u0026#39;ReadLatency\u0026#39;, \u0026#39;Dimensions\u0026#39;: [{\u0026#39;Name\u0026#39;: \u0026#39;DBInstanceIdentifier\u0026#39;, \u0026#39;Value\u0026#39;: instance_id}]}, {\u0026#39;Namespace\u0026#39;: \u0026#39;AWS/RDS\u0026#39;, \u0026#39;MetricName\u0026#39;: \u0026#39;ReadThroughput\u0026#39;, \u0026#39;Dimensions\u0026#39;: [{\u0026#39;Name\u0026#39;: \u0026#39;DBInstanceIdentifier\u0026#39;, \u0026#39;Value\u0026#39;: instance_id}]}, ] for m in metrics: resp = cw.get_metric_statistics( Namespace=m[\u0026#39;Namespace\u0026#39;], MetricName=m[\u0026#39;MetricName\u0026#39;], Dimensions=m[\u0026#39;Dimensions\u0026#39;], StartTime=start, EndTime=now, Period=60, Statistics=[\u0026#34;Average\u0026#34;], ) if df_cw_metrics.empty: df_cw_metrics=pd.DataFrame(resp[\u0026#39;Datapoints\u0026#39;]) # if \u0026#39;Unit\u0026#39; in df_cw_metrics.columns: df_cw_metrics.drop(columns=[\u0026#39;Unit\u0026#39;], inplace=True) df_cw_metrics.columns=[\u0026#39;Timestamp\u0026#39;,d_cw_col_names[m[\u0026#39;MetricName\u0026#39;]]] else: df_cw_metrics_tmp=pd.DataFrame(resp[\u0026#39;Datapoints\u0026#39;]) df_cw_metrics_tmp.drop(columns=[\u0026#39;Unit\u0026#39;], inplace=True) df_cw_metrics_tmp.columns=[\u0026#39;Timestamp\u0026#39;,d_cw_col_names[m[\u0026#39;MetricName\u0026#39;]]] df_cw_metrics = pd.merge(df_cw_metrics, df_cw_metrics_tmp, on=\u0026#39;Timestamp\u0026#39;) # 2. Performance Insights CPUUtilization df_pi_metrics=pd.DataFrame(); d_col_names={\u0026#34;CPUUtilization\u0026#34;:\u0026#34;cpuutil\u0026#34;,\u0026#34;os.cpuUtilization.total.avg\u0026#34;:\u0026#34;cputot\u0026#34;,\u0026#34;os.tasks.running.avg\u0026#34;:\u0026#34;trun\u0026#34;,\u0026#34;os.general.serverlessDatabaseCapacity.avg\u0026#34;:\u0026#34;svcap\u0026#34;,\u0026#34;os.general.maxConfiguredAcu.avg\u0026#34;:\u0026#34;maxacu\u0026#34;,\u0026#34;os.general.numVCPUs.avg\u0026#34;:\u0026#34;vcpu\u0026#34;,\u0026#34;db.User.numbackends.avg\u0026#34;:\u0026#34;nbkend\u0026#34;,\u0026#34;db.SQL.queries_started.avg\u0026#34;:\u0026#34;qstrt\u0026#34;,\u0026#34;os.diskIO.auroraStorage.diskQueueDepth.avg\u0026#34;:\u0026#34;qdepth\u0026#34;,\u0026#34;os.memory.free.avg\u0026#34;:\u0026#34;fremem\u0026#34;} pi = boto3.client(\u0026#34;pi\u0026#34;) resp = pi.get_resource_metrics( ServiceType=\u0026#34;RDS\u0026#34;, Identifier=dbi_resource_id, MetricQueries=[{\u0026#34;Metric\u0026#34;: \u0026#34;os.cpuUtilization.total.avg\u0026#34;},{\u0026#34;Metric\u0026#34;: \u0026#34;os.tasks.running.avg\u0026#34;},{\u0026#34;Metric\u0026#34;: \u0026#34;os.general.serverlessDatabaseCapacity.avg\u0026#34;} ,{\u0026#34;Metric\u0026#34;: \u0026#34;os.general.maxConfiguredAcu.avg\u0026#34;},{\u0026#34;Metric\u0026#34;: \u0026#34;os.general.numVCPUs.avg\u0026#34;},{\u0026#34;Metric\u0026#34;: \u0026#34;db.User.numbackends.avg\u0026#34;},{\u0026#34;Metric\u0026#34;: \u0026#34;db.SQL.queries_started.avg\u0026#34;} ,{\u0026#34;Metric\u0026#34;: \u0026#34;os.diskIO.auroraStorage.diskQueueDepth.avg\u0026#34;},{\u0026#34;Metric\u0026#34;: \u0026#34;os.memory.free.avg\u0026#34;} ], StartTime=start, EndTime=now, PeriodInSeconds=60, ) for metric in resp[\u0026#34;MetricList\u0026#34;]: if df_pi_metrics.empty: df_pi_metrics=pd.DataFrame(metric[\u0026#39;DataPoints\u0026#39;]) df_pi_metrics.columns=[\u0026#39;Timestamp\u0026#39;,d_col_names[metric[\u0026#34;Key\u0026#34;][\u0026#34;Metric\u0026#34;]]] else: df_pi_metrics_tmp=pd.DataFrame(metric[\u0026#39;DataPoints\u0026#39;]) df_pi_metrics_tmp.columns=[\u0026#39;Timestamp\u0026#39;,d_col_names[metric[\u0026#34;Key\u0026#34;][\u0026#34;Metric\u0026#34;]]] df_pi_metrics = pd.merge(df_pi_metrics, df_pi_metrics_tmp, on=\u0026#39;Timestamp\u0026#39;) df_pi_cw_metrics = pd.merge(df_pi_metrics, df_cw_metrics, on=\u0026#39;Timestamp\u0026#39;) print(df_pi_cw_metrics) ","date":"22 March 2026","permalink":"https://dbastreetblog.b-cdn.net/posts/post2026-01-py-script-1/","section":"Posts","summary":"","title":"Python script to display Aurora Postgres Serverless v2 cloudwatch database insights metrics"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/serverless-v2/","section":"Categories","summary":"","title":"Serverless V2"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/automation/","section":"Tags","summary":"","title":"Automation"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/aws-sdk/","section":"Categories","summary":"","title":"Aws Sdk"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/aws-sdk/","section":"Tags","summary":"","title":"Aws Sdk"},{"content":"Aws released the Aws SDK v3 for javascript in Dec 2020, with new features such as a new middleware stack and first class support for typescript . Aws has also announced end of support of Aws SDK v2 for Javascript as of Sep 2025. It is best if users were to install and use Aws SDK V3 going forward. This blog post outlines the steps to use Aws SDK V3.\nAws SDK version 3.201.0 and higher r equires Node Js 14 or higher. Below are the steps to use Aws SDK v3 to write a program that interacts with Amazon DynamoDB\nSteps to use Aws SDK V3 #Create a new project\nmkdir njsapp cd njsapp npm init npm install @aws-sdk/client-dynamodb npm install @aws-sdk/lib-dynamodb Create a node.js program\nFirst make sure that you can run .js programs as a javascript module, add the following line to your package.json file\n{ \u0026#34;name\u0026#34;: \u0026#34;njsddb\u0026#34;, \u0026#34;version\u0026#34;: \u0026#34;1.0.0\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;module\u0026#34;, \u0026#34;main\u0026#34;: \u0026#34;index.js\u0026#34;, \u0026#34;scripts\u0026#34;: { \u0026#34;test\u0026#34;: \u0026#34;echo \u0026#34;Error: no test specified\u0026#34; exit 1\u0026#34; }, \u0026#34;author\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;license\u0026#34;: \u0026#34;ISC\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;dependencies\u0026#34;: { \u0026#34;@aws-sdk/client-dynamodb\u0026#34;: \u0026#34;^3.590.0\u0026#34;, \u0026#34;@aws-sdk/lib-dynamodb\u0026#34;: \u0026#34;^3.590.0\u0026#34; } } You can use the sample code now to create a javascript program that interacts with Amazon DynamoDB. Create a .js file with the following code.\nimport { DynamoDBClient } from \u0026#34;@aws-sdk/client-dynamodb\u0026#34;; import { PutCommand, DynamoDBDocumentClient } from \u0026#34;@aws-sdk/lib-dynamodb\u0026#34;; const client = new DynamoDBClient({region:\u0026#39;us-east-2\u0026#39;}); const docClient = DynamoDBDocumentClient.from(client); export const main = async () =\u0026amp;gt; { const command = new GetCommand({ TableName: \u0026#34;ProductCatalog\u0026#34;, Key: { Id: 101, }, }); const response = await docClient.send(command); console.log(response); return response; }; main(); Make sure you add, main() as the last line in the file. You can then execute the code as follows.\nnode \u0026lt;filename\u0026gt;.js ","date":"9 June 2024","permalink":"https://dbastreetblog.b-cdn.net/posts/post2024-01-cdk-1/","section":"Posts","summary":"","title":"Configuring Aws SDK V3 for Javascript"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/dynamodb/","section":"Tags","summary":"","title":"Dynamodb"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/javascript/","section":"Categories","summary":"","title":"Javascript"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/nodejs/","section":"Tags","summary":"","title":"Nodejs"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/typescript/","section":"Categories","summary":"","title":"Typescript"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/amazon-linux-2023/","section":"Categories","summary":"","title":"Amazon Linux 2023"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/amazon-linux-2023/","section":"Tags","summary":"","title":"Amazon Linux 2023"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/client/","section":"Tags","summary":"","title":"Client"},{"content":"In order to install the postgres 15 client on Amazon Linux 2023, you can do the following. First check the available packages sudo dnf search postgres You will see in the output, a list of postgres packages, that are available to install.. In order to install the client tools, install the postgresql15 package\nsudo dnf install postgresql15 This will install the package named postgresql17.aarch64 if you are on a ARM Cpu based instance, or postgresql17.x86_64 if you are on an intel based instance Once the package is installed you have access to psql, pg_dump etc.\n","date":"15 February 2024","permalink":"https://dbastreetblog.b-cdn.net/posts/post2024-02-postgres15-1/","section":"Posts","summary":"","title":"Installing postgres 15 client on Amazon Linux 2023"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/postgres-15/","section":"Categories","summary":"","title":"Postgres 15"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/postgres-15/","section":"Tags","summary":"","title":"Postgres 15"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/amazon-linux-2/","section":"Categories","summary":"","title":"Amazon Linux 2"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/amazon-linux-2/","section":"Tags","summary":"","title":"Amazon Linux 2"},{"content":"If you are using a Amazon Linux 2 based Ec2 instance, to install and run postgresql tools like psql and pg_dump, today the highest versions of postgres tools available from the repository is postgres 14. You can check the currently available versions using the following command\namazon-linux-extras | grep -i postgre This command will list versions postgres 10 through 14.\nYou can enable the Redhat repositories for postgres on your amazon linux 2 host, and install the postgres 15.x client from those repositories. The steps are as follows.\nUpgrade to the latest version of amazon linux 2 (Optional)\nIf you already have a postgres package installed consider removing it.\nEnable the postgres repo for redhat linux\ncreate a new file /etc/yum.repos.d/pgdg.repo\nAdd the following lines to pgdg.repo\n[pgdg15] name=PostgreSQL 15 for Redhat Linux - x86_64 baseurl=https://download.postgresql.org/pub/repos/yum/15/redhat/rhel-7.10-x86_64 enabled=1 gpgcheck=0 sudo yum makecache sudo yum repolist (ensure that pgdg15 shows as a repository) sudo yum install libzstd sudo yum --disablerepo=\u0026#34;*\u0026#34; --enablerepo=\u0026#34;pgdg15\u0026#34; install postgresql15-15.3-1PGDG.rhel7 (To get postgres 15.3) Now you should have 15.3 versions of psql and pg_dump etc\n","date":"3 September 2023","permalink":"https://dbastreetblog.b-cdn.net/posts/post2023-01-postgres15-1/","section":"Posts","summary":"","title":"Installing the postgres 15.x client on Amazon Linux 2"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/postgres/","section":"Categories","summary":"","title":"Postgres"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/database/","section":"Categories","summary":"","title":"Database"},{"content":"Using the postgres docker image, is the quickest way to get started with any of your postgres development environments, or if you need to test any features of the postgres database. Official postgres database images are available for download and use from docker hub. You can get images of postgres 11,12,13,14 and 15 from docker hub. In this blog post we will see how to get started with downloading and running this image on Ubuntu Linux.\nThe steps outlined below assume that you have one of the newer versions of ubuntu installed on a host and that you are using the linux user named “ubuntu” to run docker.\nAs root execute the following commands on the ubuntu host to download docker. These steps also setup the ubuntu user to run docker containers.\nmkdir -m 0755 -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg echo \\ \u0026quot;deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \\ $(lsb_release -cs) stable\u0026quot; | sudo tee /etc/apt/sources.list.d/docker.list \u0026gt; /dev/null apt-get update -y apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y systemctl disable --now docker.service docker.socket su - ubuntu -c \u0026quot;dockerd-rootless-setuptool.sh install\u0026quot; Now you are ready to download and run the postgres container.\nLogin as the “ubuntu” user and Create a text file named pg.yaml file, that has the following instructions. The following instructions assume that you have a directory /u01/pg/data which you want to use to store your postgres datafiles.\nservices: db: image: postgres:latest restart: always environment: POSTGRES_PASSWORD: MyPassword ports: - \u0026quot;5432:5432\u0026quot; volumes: - /u01/pg/data:/var/lib/postgresql/data Now we can run docker to download and run this image using the following command\ndocker compose -f /home/ubuntu/tmp/pg.yaml up -d db This command downloads and installs the latest postgres image (Which at the time of writing is 15.2) You can now connect to the postgres database from the ubuntu host as shown below.\npsql -h localhost -p 5432 -d postgres -U postgres You can login as root to your postgres container using the command below\ndocker exec -it cc18a27f8850 bash where cc18a27f8850 is the container id (You can get your container id by running the command,\ndocker container ls -a. Once you are logged in as root to the postgres container, you can su – postgres and then run commands to stop and start postgres. pg_ctl can be found in /usr/lib/postgresql/15/bin.\n","date":"18 April 2023","permalink":"https://dbastreetblog.b-cdn.net/posts/post2023-01-pg-docker-1/","section":"Posts","summary":"","title":"Installing and using PostgreSql docker image"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/imdb/","section":"Tags","summary":"","title":"Imdb"},{"content":"IMDB (Internet Movie Database) makes the movie dataset available for free download at https://datasets.imdbws.com/. The documentation for this dataset can be found at https://www.imdb.com/interfaces/.\nIn this blog post, I show you how to load this data into a PostgreSql database. The steps are executed from an Ubuntu Linux workstation. I assume that you already have a postgresql database with about 50Gb of free space to upload this data into and you know the connection information.\nWe\u0026rsquo;ll use the s32cinemagoer.py script that can be downloaded from github.\nFrom the ubuntu workstation, install the following packages\nsudo apt install python3-pip sudo apt-get install libpq-dev Now install the following python modules\npip3 install Psycopg2 pip3 install imdbpy Create a new directory, to store the downloaded datafiles and download the datafiles from https://datasets.imdbws.com into this directory\nmkdir dat cd dat wget https://datasets.imdbws.com/name.basics.tsv.gz wget https://datasets.imdbws.com/title.akas.tsv.gz wget https://datasets.imdbws.com/title.basics.tsv.gz wget https://datasets.imdbws.com/title.crew.tsv.gz wget https://datasets.imdbws.com/title.episode.tsv.gz wget https://datasets.imdbws.com/title.principals.tsv.gz wget https://datasets.imdbws.com/title.ratings.tsv.gz Login to your postgresql database and create a new schema to hold the imdb tables (This step is optional. If you do not create this schema, then the tables and the corresponding data gets loaded into the public schema).\ncreate schema imdb; Before you run the script, lets edit the script and make one change, which will enable the script to load the data into the newly created imdb schema. This change will be made in line 183 in the file.\nChange engine = sqlalchemy.create_engine(db_uri, encoding=\u0026#39;utf-8\u0026#39;, echo=False) To engine = sqlalchemy.create_engine(db_uri, echo=False) Change metadata.create_all(tables=[table]) To metadata.create_all(bind=metadata.bind,tables=[table]) Then Execute the script as shown below\npython3 s32cinemagoer.py /home/ubuntu/dat postgresql://username:password@dbhostname/dbname Where /home/ubuntu/dat is the directory where the imdb files are downloaded into.\nThis will take some time to load (Close to an hour on a reasonably sized ubuntu workstation) and will consume about 15Gb of space in your postgresql database.\n","date":"18 March 2023","permalink":"https://dbastreetblog.b-cdn.net/posts/post2023-02-postgres-imdb/","section":"Posts","summary":"","title":"Loading IMDB data into postgresql"},{"content":"Elasticsearch is an opensource text search engine. It is accessible using RESTFul API\u0026rsquo;s and uses JSON documents to store data. It allows users to search very large amounts of data at a very high speed. Since it is written using the Java programming language it can run on many platforms. See this blog post on how to install, configure and start Elasticsearch 8.5.0.\nIn this blog post I will show you how you can create an index, insert data into the index and query the data in Elasticsearch. In the rest of this post, I assume that you are executing the curl commands from the host that is running the Elasticsearch server. Hence all the curl queries are directed to localhost. If you are running the curl commands from a client machine, replace localhost with your actual hostname (or ip address) for the server running Elasticsearch.\nInserting a single document into an Index\ncurl -k -POST https://localhost:9200/messages/_doc -H \u0026#34;Content-Type: application/json\u0026#34; -d\u0026#39;{ \u0026#34;msg_id\u0026#34; : 1, \u0026#34;msg_case_id\u0026#34; : 55, \u0026#34;msg_message\u0026#34; : \u0026#34;Hi This is xyz from def media company\u0026#34;}\u0026#39; --user \u0026#34;elastic:Password1\u0026#34; In the command shown above, \u0026ldquo;messages\u0026rdquo; is the Index that we are inserting the document into. We are also providing the Elasticsearch username and password to set the POST requirement to Elasticsearch. The -k is used to tell curl to accept the selft signed certificated from Elasticsearch.\nBulk inserting documents into an Index\nFirst create a file named msg.json with the following lines\n{\u0026#34;index\u0026#34;:{\u0026#34;_id\u0026#34;:\u0026#34;1\u0026#34;}}{ \u0026#34;msg_id\u0026#34; : 1, \u0026#34;msg_case_id\u0026#34; : 55, \u0026#34;msg_message\u0026#34; : \u0026#34;Hi This is xyz from def media company\u0026#34; }{\u0026#34;index\u0026#34;:{\u0026#34;_id\u0026#34;:\u0026#34;2\u0026#34;}} { \u0026#34;msg_id\u0026#34; : 2, \u0026#34;msg_case_id\u0026#34; : 55, \u0026#34;msg_message\u0026#34; : \u0026#34;We provide targeted advertising to different platforms\u0026#34; }{\u0026#34;index\u0026#34;:{\u0026#34;_id\u0026#34;:\u0026#34;3\u0026#34;}}{ \u0026#34;msg_id\u0026#34; : 3, \u0026#34;msg_case_id\u0026#34; : 55, \u0026#34;msg_message\u0026#34; : \u0026#34;Includes TV, Radio, Online, Social Media etc\u0026#34; }{\u0026#34;index\u0026#34;:{\u0026#34;_id\u0026#34;:\u0026#34;4\u0026#34;}}{ \u0026#34;msg_id\u0026#34; : 4, \u0026#34;msg_case_id\u0026#34; : 55, \u0026#34;msg_message\u0026#34; : \u0026#34;Our conversion ratios are very high\u0026#34; }{\u0026#34;index\u0026#34;:{\u0026#34;_id\u0026#34;:\u0026#34;5\u0026#34;}}{ \u0026#34;msg_id\u0026#34; : 5, \u0026#34;msg_case_id\u0026#34; : 55, \u0026#34;msg_message\u0026#34; : \u0026#34;provides search engine optimization\u0026#34; } You can batch insert all the documents in the file above to the Elasticsearch index named \u0026ldquo;messages\u0026rdquo; using the command below\ncurl -k -X POST https://localhost:9200/messages/_bulk?pretty -H \u0026#34;Content-Type: application/x-ndjson\u0026#34; --user \u0026#34;elastic:Password1\u0026#34; --data-binary @msg.json Note that we are using the content-type x-ndjson here\nYou can delete all the documents in the \u0026ldquo;messages\u0026rdquo; index, using the command below\ncurl -k -XPOST \u0026#39;https://localhost:9200/messages/_delete_by_query?conflicts=proceed\u0026amp;amp;pretty\u0026#39; -H \u0026#39;Content-Type: application/json\u0026#39; --user \u0026#34;elastic:Password1\u0026#34; -d\u0026#39;{\u0026#34;query\u0026#34;: {\u0026#34;match_all\u0026#34;: {}}}\u0026#39; Querying documents\nYou can query all the documents from the \u0026ldquo;messages\u0026rdquo; index using the command below\ncurl -k -X GET \u0026#39;https://localhost:9200/messages/_search\u0026#39; --user \u0026#34;elastic:Password1\u0026#34; You can query specific documents that match specific criteria using the command below\ncurl -k -X POST \u0026#34;https://localhost:9200/messages/_search?pretty\u0026#34; -H \u0026#39;Content-Type: application/json\u0026#39; --user \u0026#34;elastic:Password1\u0026#34; -d\u0026#39;{\u0026#34;query\u0026#34;: {\u0026#34;bool\u0026#34;: {\u0026#34;filter\u0026#34;: [{\u0026#34;term\u0026#34;: {\u0026#34;msg_message\u0026#34;: \u0026#34;ratio\u0026#34;}},{\u0026#34;term\u0026#34;: {\u0026#34;msg_id\u0026#34;: \u0026#34;4\u0026#34;}}]}}}\u0026#39; The above query will display the documents where the word ratio occurs in the msg_message property and when the msg_id property is 4.\nIn this blog post I have shown you how to insert documents into Elasticsearch indexes and query them.\n","date":"10 November 2022","permalink":"https://dbastreetblog.b-cdn.net/posts/post2023-02-elasticsearch-2/","section":"Posts","summary":"","title":"Adding documents  and querying Elasticsearch"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/elasticsearch/","section":"Categories","summary":"","title":"Elasticsearch"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/elasticsearch/","section":"Tags","summary":"","title":"Elasticsearch"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/insert/","section":"Tags","summary":"","title":"Insert"},{"content":"Elasticsearch is an opensource search engine based on the Lucene library and Kibana is an opensource data visualization dashboard for Elasticsearch. In this post I will show you how to install Elasticsearch and Kibana on a virtual machine (In this case running the Amazon Linux 2 Operating System).\nInstall the required packages #sudo yum update -y sudo yum group install \u0026#34;Development Tools\u0026#34; -y yum install wget -y yum install readline-devel -y yum install openssl-devel -y Install ElasticSearch #curl -O \u0026#34;https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.5.0-linux-x86_64.tar.gz\u0026#34; tar -xvf elasticsearch-8.5.0-linux-x86_64.tar.gz chown -R ec2-user:ec2-user /home/ec2-user/elasticcd /home/ec2-user/elastic/elasticsearch-8.5.0/config Add the following lines to elasticsearch.yml #transport.host: localhosttransport.port: 9300 http.port: 9200 network.host: 0.0.0.0 Install Kibana #cd /home/ec2-user/ curl -O \u0026#34;https://artifacts.elastic.co/downloads/kibana/kibana-8.5.0-linux-x86_64.tar.gz\u0026#34; tar -xzf kibana-8.5.0-linux-x86_64.tar.gz chown -R ec2-user:ec2-user /home/ec2-user/kibana-8.5.0 cd /home/ec2-user/kibana-8.5.0/config Add the following lines to kibana.yml\nserver.host=\u0026lt;your-host-ip-address\u0026gt; In the line shown above, make sure to replace with your hosts actual ip address.\nStart Elasticsearch and Kibana #cd /home/ec2-user/elastic/elasticsearch-8.5.0/bin nohup ./elasticsearch -Epath.data=data -Epath.logs=log \u0026amp; cd /home/ec2-user/kibana-8.5.0/binnohup ./kibana \u0026amp;\nSetup a new password for the Elasticsearch superuser #cd /home/ec2-user/elastic/elasticsearch-8.5.0bin/elasticsearch-reset-password -u elastic \u0026ndash;interactive\nfollow the prompts to setup a new password for the user elastic.\nAt this time elasticsearch has been installed and started and you can start creating your indexes and running your queries.\n","date":"10 November 2022","permalink":"https://dbastreetblog.b-cdn.net/posts/post2022-03-elasticsearch-3/","section":"Posts","summary":"","title":"Installing Elasticsearch and Kibana 8.5.0"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/replication/","section":"Tags","summary":"","title":"Replication"},{"content":"When postgresql is used as a transactional database, there are use cases where the data changes from the transactional database are captured and send to other databases like your datamart or datawarehouse. You could use cloud services like Aws Dms or replication software like Debazium to do this. In this blog post I will show you how to use python to read changes (cdc, change data capture) from a postgresql database using the wal2json output plugin and psycopg2.\nWhen you are compiling postgresql from source code, you can enable the output plugins test_decoder or wal2json as shown below.\ncd /home/postgres/tmp/postgresql-14.4/contrib/test_decoding make PG_CONFIG=/u01/pg/14/bin/pg_config make PG_CONFIG=/u01/pg/14/bin/pg_config install cd /home/postgres/tmp/postgresql-14.4/ wget https://github.com/eulerto/wal2json/archive/refs/tags/wal2json_2_4.tar.gz tar -xzvf wal2json_2_4.tar.gz cd wal2json-wal2json_2_4/ make PG_CONFIG=/u01/pg/14/bin/pg_config make PG_CONFIG=/u01/pg/14/bin/pg_config install where /home/postgres/tmp/postgresql-14.4 is the directory where you untarred your postgres source code into, before compiling and installing postgres to /u01/pg/14 In order to proceed you need to have installed the library psycopg2 with python3 (Eg: pip install psycopg2) You also need to make sure that the parameter wal_level is set to ‘logical’ in the postgresql.conf file of your postgres database. create a table named books in your postgresql database.\ncreate table books (bookid bigint primary key,bookname varchar(100)); Insert a few rows into the table.\ninsert into books values (1,'First Book'); insert into books values (2,'Second Book'); insert into books values (3,'Third Book'); The python library psycopg2 has a module named extras which provides helpers to read from postgres logical replication publishers. We will be using the functions from this module, namely create_replication_slot , start_replication and consume_stream to create the publisher and subscriber for logical replication.\nHere is the code sample for pglogical.py\nfrom __future__ import print_function import sys import psycopg2 import psycopg2.extras conn = psycopg2.connect( 'host=localhost user=postgres port=5432', connection_factory=psycopg2.extras.LogicalReplicationConnection) cur = conn.cursor() replication_options = { 'include-xids':'1', 'include-timestamp':'1', 'pretty-print':'1' } try: cur.start_replication( slot_name='pytest', decode=True, options=replication_options) except psycopg2.ProgrammingError: cur.create_replication_slot('pytest', output_plugin='wal2json') cur.start_replication( slot_name='pytest', decode=True, options=replication_options) class DemoConsumer(object): def __call__(self, msg): print(msg.payload) msg.cursor.send_feedback(flush_lsn=msg.data_start) democonsumer = DemoConsumer() print(\u0026quot;Starting streaming, press Control-C to end...\u0026quot;, file=sys.stderr) try: cur.consume_stream(democonsumer) except KeyboardInterrupt: cur.close() conn.close() print(\u0026quot;The slot 'pytest' still exists. Drop it with \u0026quot; \u0026quot;SELECT pg_drop_replication_slot('pytest'); if no longer needed.\u0026quot;, file=sys.stderr) print(\u0026quot;WARNING: Transaction logs will accumulate in pg_xlog \u0026quot; \u0026quot;until the slot is dropped.\u0026quot;, file=sys.stderr) The code above is a modified version of the code, published by Marco Nenciarini here. The code above uses the wal2json output plugin, you can change it to use the test_decoding output plugin, but you will also have to change the replication_options variable to those supported by test_decoding. You can run the following command to create the publication and the subscriber.\npython3 pglogical.py You will see a prompt saying “Starting streaming, press Control-C to end…” Let us now make a few changes to the books table.\nnsert into books values (5,\u0026lsquo;Fifth Book\u0026rsquo;);\ndo $$ \u0026lt;\u0026lt;first_block\u0026gt;\u0026gt; begin insert into books values (6,'Sixth Book'); delete from books where bookid = 3; end first_block $$; If you now go back to the screen where you ran your python program, you can see the following messages on screen.\n{ \u0026quot;xid\u0026quot;: 741, \u0026quot;timestamp\u0026quot;: \u0026quot;2022-07-11 21:29:37.301299+00\u0026quot;, \u0026quot;change\u0026quot;: [ { \u0026quot;kind\u0026quot;: \u0026quot;insert\u0026quot;, \u0026quot;schema\u0026quot;: \u0026quot;public\u0026quot;, \u0026quot;table\u0026quot;: \u0026quot;books\u0026quot;, \u0026quot;columnnames\u0026quot;: [\u0026quot;bookid\u0026quot;, \u0026quot;bookname\u0026quot;], \u0026quot;columntypes\u0026quot;: [\u0026quot;bigint\u0026quot;, \u0026quot;character varying(100)\u0026quot;], \u0026quot;columnvalues\u0026quot;: [5, \u0026quot;Fifth Book\u0026quot;] } ] } { \u0026quot;xid\u0026quot;: 742, \u0026quot;timestamp\u0026quot;: \u0026quot;2022-07-11 21:33:07.913942+00\u0026quot;, \u0026quot;change\u0026quot;: [ { \u0026quot;kind\u0026quot;: \u0026quot;insert\u0026quot;, \u0026quot;schema\u0026quot;: \u0026quot;public\u0026quot;, \u0026quot;table\u0026quot;: \u0026quot;books\u0026quot;, \u0026quot;columnnames\u0026quot;: [\u0026quot;bookid\u0026quot;, \u0026quot;bookname\u0026quot;], \u0026quot;columntypes\u0026quot;: [\u0026quot;bigint\u0026quot;, \u0026quot;character varying(100)\u0026quot;], \u0026quot;columnvalues\u0026quot;: [6, \u0026quot;Sixth Book\u0026quot;] } ,{ \u0026quot;kind\u0026quot;: \u0026quot;delete\u0026quot;, \u0026quot;schema\u0026quot;: \u0026quot;public\u0026quot;, \u0026quot;table\u0026quot;: \u0026quot;books\u0026quot;, \u0026quot;oldkeys\u0026quot;: { \u0026quot;keynames\u0026quot;: [\u0026quot;bookid\u0026quot;], \u0026quot;keytypes\u0026quot;: [\u0026quot;bigint\u0026quot;], \u0026quot;keyvalues\u0026quot;: [3] } } ] }\nThese are the changes the python subscriber program is reading from postgres the logical replication publisher. You can then write these changes either to a csv file or to another database as you choose.\n","date":"12 July 2022","permalink":"https://dbastreetblog.b-cdn.net/posts/post2022-01-pg-replication-1/","section":"Posts","summary":"","title":"Subscribing to postgresql replication using python psycopg2"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/test_decoding/","section":"Tags","summary":"","title":"Test_decoding"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/installation/","section":"Tags","summary":"","title":"Installation"},{"content":"pg_partman is an extension that simplifies the process of partition management in postgres. Below are the steps that I followed to install the pg_partman extension with postgres 12.8. Change your current working directory to the directory where you unzipped the postgres 12.8 source code.\ncd \u0026lt;SomeStaticPath\u0026gt;/postgresql-12.8/contrib git clone https://github.com/pgpartman/pg_partman.git cd pg_partman make PG_CONFIG=\u0026lt;pghome\u0026gt;/bin/pg_config NO_BGW=1 make install Then Edit your postgresql.conf file and add pg_partman_bgw to the parameter shared_preload_libraries Now restart your postgres instance At this time you are ready to create the extension from postgres and use it. Use psql to login to your database\nCREATE SCHEMA partman; CREATE EXTENSION pg_partman SCHEMA partman; \\dx (To list the extensions and the version installed in your database) ","date":"29 November 2021","permalink":"https://dbastreetblog.b-cdn.net/posts/post2021-01-pg-partman-1/","section":"Posts","summary":"","title":"Installing the pg_partman extension"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/pg_partman/","section":"Tags","summary":"","title":"Pg_partman"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/psql/","section":"Tags","summary":"","title":"Psql"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/substitution-variable/","section":"Tags","summary":"","title":"Substitution Variable"},{"content":"As postgresql users and administrators, we tend to create lot of scripts, and run them routinely, for common queries we need to execute against the database. It is likely that in some of those scripts, you would want to parameterize values used in filter conditions. Below is an example of how this can be done.\npsql is the terminal based front end tool to interact with postgresql. You can run scripts stored in files in the file system, using the \\i directive in plsql. You can use the \\set command to set variables in psql. If you want to prompt for the value to be entered by the user, you can use the \\prompt directive.\nCreate a file pgstats.sql with the following lines.\nprompt \u0026#39;Enter Table Name : \u0026#39; tabname select last_vacuum,last_autovacuum,last_analyze ,last_autoanalyze,n_live_tup,n_dead_tupfrom pg_stat_user_tableswhere relname = :\u0026#39;tabname\u0026#39;; Now if you run this from psql you will be first prompted for the table name and then it will display the results for the table name you entered.\npostgres=# \\i pgstats.sql Enter Table Name : nflstats last_vacuum | last_autovacuum | last_analyze | last_autoanalyze | n_live_tup | n_dead_tup -------------+-----------------+------------------------------+-------------------------------+------------+------------ | | 2020-06-08 18:07:36.29538+00 | 2020-06-08 15:37:48.738104+00 | 270418 | 0 ","date":"9 June 2020","permalink":"https://dbastreetblog.b-cdn.net/posts/post2020-01-psql-1/","section":"Posts","summary":"","title":"Substitution variables in psql scripts"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/aws/","section":"Categories","summary":"","title":"Aws"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/tags/aws-cli/","section":"Tags","summary":"","title":"Aws Cli"},{"content":"","date":null,"permalink":"https://dbastreetblog.b-cdn.net/categories/cloudwatch/","section":"Categories","summary":"","title":"Cloudwatch"},{"content":"The following commands can be used to list the Error messages from cloudwatch logs, produced from DMS (Database Migration service). First list the log groups\naws logs describe-log-groups Next list the log streams in your log group\naws logs describe-log-streams --log-group-name \u0026amp;lt;YourLogGroupNameHere\u0026gt; Next list the error messages. Within the DMS log, the Errors are indicated with a pattern \u0026ldquo;E:\u0026rdquo; within the error string, so that is the pattern we search for.\naws logs filter-log-events --log-group-name \u0026amp;lt;YourLogGroupNameHere\u0026gt; --log-stream-names \u0026amp;lt;YourLogStreamNameHere\u0026gt; --filter-pattern \u0026#34;[message = \u0026#34;*E:*\u0026#34;]\u0026#34; --query \u0026#39;events[*].message\u0026#39; If you are searching in cloudwatch logs produced from other sevices, you should replace E: with the pattern that flags Error messages for that service.\n","date":"14 April 2020","permalink":"https://dbastreetblog.b-cdn.net/posts/post2020-02-awscli-1/","section":"Posts","summary":"","title":"Listing Errors from Cloudwatch logs using Aws Cli"},{"content":"My name is Rajeev Thottathil and I am a database specialist Solutions Architect with 33 years experience architecting and implementing mission critical,resilient databases at scale on-premises and in the cloud. I mostly work with Oracle, Postgres, Mysql,DynamoDb,Redis,Neptune and DocumentDb databases. In this blog which i started in 2008 I publish articles regarding things i learn about databases, linux, python etc which may come in handy for others.\nThis blog has always been a labor of love, writing about things i work on and wanted to share with others. In 2008 it used to run on a AMD Athlon based computer (bought from a custom computer builder pcsforeveryone in cambridge, MA) running Ubuntu Linux (Self installed mysql and wordpress) . Over the years it has moved from being hosted at godaddy to ipage to hostinger. There were brief periods of hiatus too. Every time the hosting provider jacked up the prices i move it. Now this site runs using hugo (Fairly easy to create and edit markdown files) as a static site, with the content served from bunny.net.\nDisclaimer : All opinions in this blog are mine and does not reflect the opinions of any of my current or past employers. The samples provided in this blog are only to be used for educational purposes and not for production deployments.\n","date":"22 June 2008","permalink":"https://dbastreetblog.b-cdn.net/about/","section":"dbaStreet Blog","summary":"","title":"About Me"}]