Category Archives: open source

A Good Database BUG

People generally do not think of cockroaches positively, but I have nothing but good feelings about CockroachDB. At its core, CockroachDB is resilient and reliable.

Cockroach Labs, a software company known for its cloud-native SQL databases, has found a home in Bengaluru, India. With a rapidly growing team of over 55 engineers specializing in database and cloud engineering, the company’s journey in India is as much about emotional ties as it is about strategic growth.

Bengaluru’s choice is strategic. It offers unparalleled time zone advantages and access to a rich talent pool. With a population of 1.4 billion and a digitizing economy, India is ideal for testing CockroachDB’s resilience and scalability.

The company plans to expand its Bengaluru office into a first-class R&D hub. Teams are working on innovations like vector data integration for AI, enabling operational databases to evolve into systems capable of real-time intelligence.

Building Blocks of CockroachDB

The founders’ lack of a transactional distributed database forced them to use DynamoDB, leading to inefficiencies in their early startup years. This frustration led to the birth of Cockroach Labs in 2014, with a vision to create an open-source, cloud-native distributed database.


I am a HUGE advocate of open-source databases, so this journey is intriguing. Not sitting with inefficiencies but finding a way to grow beyond them is a significant step for any startup.

True to its name, CockroachDB has built a reputation for resilience. It can run seamlessly across cloud providers, private data centers, and hybrid setups, making it a standout choice. Cockroach Labs focuses on eliminating vendor lock-in and ensuring businesses can operate uninterrupted, even during cloud or data center outages. I can’t say enough how important it is not to be locked into one cloud provider. This is a serious flex for an open-source database NOT to be “vendor dependent.” Staying in the driver’s seat and not becoming a passenger or going along for a ride with a service provider is ideal. Retaining the power of “choice” as a customer is priceless. This adaptability has made Cockroach Labs the operational backbone for global giants like Netflix and ambitious startups like Fi.

Sharing some notes on my explorer experience:

Getting Started

Install CockroachDB on Ubuntu (using Bash Shell):

1. Update Your System: First, update your system packages to the latest version:
   
   sudo apt update -y
   sudo apt upgrade -y
   

2. Install Dependencies: Install the required dependencies:
   
   sudo apt install -y apt-transport-https ca-certificates curl software-properties-common
  

3. Download CockroachDB: Download the latest version of CockroachDB:
   
   wget -qO- https://binaries.cockroachdb.com/cockroach-latest.linux-amd64.tgz | tar xvz
   

4. Move CockroachDB Binary: Move the binary to a directory in your PATH:

      sudo cp -i cockroach-latest.linux-amd64/cockroach /usr/local/bin/
   

5. Verify Installation: Verify the installation by checking the CockroachDB version:
   
   cockroach version
   

6. Initialize CockroachDB Cluster: Create a directory for CockroachDB data and initialize the cluster:
   
   sudo mkdir -p /var/lib/cockroach
   sudo chown $(whoami) /var/lib/cockroach
   cockroach start-single-node --insecure --store=/var/lib/cockroach --listen-addr=localhost:26257 --http-addr=localhost:8080
   

7. Connect to CockroachDB SQL Shell: Connect to the CockroachDB SQL shell:
   
   cockroach sql --insecure --host=localhost:26257
   

8. Run CockroachDB as a Background Service: Create a systemd service file to run CockroachDB as a background service:
   
   sudo nano /etc/systemd/system/cockroach.service
   
   Add the following configuration:
   ini
   [Unit]
   Description=CockroachDB
   Documentation=https://www.cockroachlabs.com/docs/

   [Service]
   Type=notify
   ExecStart=/usr/local/bin/cockroach start-single-node --insecure --store=/var/lib/cockroach --listen-addr=localhost:26257 --http-addr=localhost:8080
   TimeoutStartSec=0
   Restart=always
   RestartSec=10

   [Install]
   WantedBy=multi-user.target
   

9. Enable and Start the Service: Reload the systemd manager configuration, start the CockroachDB service, and enable it to run on system startup:
   
   sudo systemctl daemon-reload
   sudo systemctl start cockroach
   sudo systemctl enable cockroach
   sudo systemctl status cockroach
   
CockroachDB is now installed and running on your Ubuntu system. 

Cockroach Labs is continuing to invests heavily in AI-specific features, including support for vector similarity searches and operationalizing AI workflows.

Here's an example of how you can use CockroachDB with AI, specifically leveraging vector search for similarity searches:

1. Install CockroachDB: Follow the steps I provided earlier to install CockroachDB on your system.

2. Create a Database and Table: Connect to CockroachDB and create a database and table to store your data:
 
   cockroach sql --insecure --host=localhost:26257
   CREATE DATABASE ai_example;
   USE ai_example;
   CREATE TABLE vectors (id INT PRIMARY KEY, vector FLOAT[] NOT NULL);
 

3. Insert Data: Insert some sample data into the table:(at sql prompt) Steps 3 -5

   INSERT INTO vectors (id, vector) VALUES (1, ARRAY[1.0, 2.0, 3.0]), (2, ARRAY[4.0, 5.0, 6.0]);


4. Enable pgvector Extension**: Enable the `pgvector` extension for vector similarity searches:
sql>
   CREATE EXTENSION IF NOT EXISTS pgvector;


5. Perform Similarity Search**: Use the `pgvector` extension to perform a similarity search:
sql>
   SELECT id, vector, similarity(vector, ARRAY[2.0, 3.0, 4.0]) AS similarity_score
   FROM vectors
   ORDER BY similarity_score DESC;
 

Create a table to store vectors, and perform similarity searches using the `pgvector` extension.

 "pgvector" enables similarity searches by comparing high-dimensional vectors, making it useful for tasks like finding similar items in recommendation systems, which is an AI tool. 

Yes. CockroachDB is compatible with PostgreSQL, which means you can use many PostgreSQL tools, libraries, and client applications. This can be a bridge in learning about this database, which is also a plus.

I am looking forward to testing these new developments from Cockroach Labs. There is a wealth information contained in their repository (linked-below) as well as number of repos from the open-source database community. Their investment in AI is key to the company’ sustainable growth.

https://github.com/cockroachlabs

https://www.cockroachlabs.com

Learn more about pgvector in this repo

Simplifying SQL Generation with Vanna: An Open-Source Python RAG Framework


Handling databases often involves crafting complex SQL queries, which can be daunting for those who aren’t SQL experts. The need for a user-friendly solution to streamline SQL generation has led to the development of Vanna, an open-source Python framework.

The Challenge


Crafting complex SQL queries can be time-consuming and requires a deep understanding of the database structure. Existing methods might assist but often lack adaptability to various databases or compromise privacy and security.

Introducing Vanna


Vanna uses a Retrieval-Augmented Generation (RAG) model to take a unique two-step approach.

How it Works – In Two Steps (clicks)

First, users train the model on their data, and then they can pose questions to obtain SQL queries tailored to their specific database.

Key Features


Simplicity and Versatility: Vanna stands out for its simplicity and adaptability. Users can train the model using Data Definition Language (DDL) statements, documentation, or existing SQL queries, allowing for a customized and user-friendly training process.

Direct Execution:

Vanna processes user queries and returns SQL queries that are ready to be executed on the database. This eliminates the need for intricate manual query construction, providing a more accessible way to interact with databases.

High Accuracy

Vanna excels in accuracy, particularly on complex datasets. Its adaptability to different databases and portability across Language Model Models (LLMs) make it a cost-effective and future-proof solution.

Security Measures

Operating securely, Vanna ensures that database contents stay within the local environment, prioritizing privacy.

Continuous Improvement


Vanna supports a self-learning mechanism. In Jupyter Notebooks, it can be set to “auto-train” based on successfully executed queries. Other interfaces prompt users for feedback and store correct question-to-SQL pairs for continual improvement and enhanced accuracy.

Flexible Front-End Experience


Whether working in Jupyter Notebooks or extending functionality to end-users through platforms like Slackbot, web apps, or Streamlit apps, Vanna provides a flexible and user-friendly front-end experience.

Vanna addresses the common pain point of SQL query generation by offering a straightforward and adaptable solution. Its metrics underscore its accuracy and efficiency, making it a valuable tool for working with databases, regardless of SQL expertise. With Vanna, querying databases becomes more accessible and user-friendly.

As an Engineer who loves working with data, I am looking forward to trying Vanna to level up my SQL development.

GitHub

Documentation

Image Credit: Vanna AI

Claude 2.1 Lets Go!

One of the things I appreciate and respect about Anthropic, the creators of Claude, is the transparency of their messaging and content. The content is easy to understand, and that’s a plus in this space. Whenever I visit their site, I have a clear picture of where they are and the plans for moving forward. OpenAI’s recent shenanigans have piqued my curiosity to revisit other chatbot tools. Over a month ago, I wrote a comparative discussion about a few AI tools. One of the tools I discussed was Claude 2.0. Now that Claude 2.1 has been released, I wanted to share a few highlights based on my research. Note most of these features are by invitation only (API Console)or fee-based (Pro Access only) and are not generally available now in the free tier. There is a robust documentation library for Claude to review.

The Basics

  • Claude 2.1 is a chatbot tool developed by Anthropic. The company builds large language models (LLM) as a cornerstone of its development initiatives and its flagship chatbot, Claude.
  • Claude 2.1 manages the API console in Anthropics’s latest release. This AI machine powers the claude.ai chat experience.
  • In the previous version, Claude 2.0 could handle 100,000 tokens that translated to inputs of around 75,000 words.
  • A token is a unit measurement of text AI models use to represent and process natural language. The unit can be code, text, or characters, depending on the method of tokenization used. The unit of text is assigned a numeric value fed into the model.
  • Claude 2.1 delivers an industry-leading 200K token context window, translating to around 150,000 words, or about 500 pages.
  • A significant reduction in rates of model hallucination and system prompts in version 2.1 means more consistent and accurate responses.

200k Tokens Oh My!

Why the increase in the number of tokens? Anthropic is listening to their growing community of users. Based on use cases, Claude was used for application development and analyzing complex plans and documents. Users wanted more tokens to review large data sets. Claude aims to produce more accurate outputs when working with larger data sets and longer documents.

With this increase in tokens, users can now upload technical documentation like entire codebases, technical documentation, or financial reports. By analyzing detailed content or data, Claude can summarize, conduct Q&A, forecast trends, spot variations across several revisions of the same content, and more.

Processing large datasets and leveraging the benefits of AI by pushing the limit up to 200,000 tokens is a complex feat and an industry first. Although AI cannot replace humans altogether, it can allow humans to use time more efficiently. Tasks typically requiring hours of human effort to complete may take Claude a few minutes. Latency should decrease substantially as this type of technology progresses.

Decrease in Hallucination Rates

Although I am interested in the hallucination aspects of AI, for most this is not ideal in business. Claude 2.1 has also made significant gains in credibility, with a decrease in false statements compared to the previous Claude 2.0 model. Companies can build high-performing AI applications that solve concrete business problems and deploy AI with the goal of greater trust and reliability.

Claude 2.1 has also made meaningful improvements in comprehension and summarization, particularly for long, complex documents that demand high accuracy, such as legal documents, financial reports, and technical specifications. Use cases have shown that Claude 2.1 demonstrated more than a 25% reduction in incorrect answers and a 2x or lower rate of mistakenly concluding a document supports a particular claim. Claude continues to focus on enhancing their outputs’ precision and dependability.

API Tool Use

I am excited to hear about the beta feature that allows Claude to integrate with users’ existing processes, products, and APIs. This expanded interoperability aims to make Claude more useful. Claude can now orchestrate across developer-defined functions or APIs, search over web sources, and retrieve information from private knowledge bases. Users can define a set of tools for Claude and specify a request. The model will then decide which device is required to achieve the task and execute an action on its behalf.

The Console

New consoles can often be overwhelming, but Claude made the commendable choice to simplify their developer Console experience for Claude API users while making it easier to test new prompts for faster learning. The new Workbench product will enable developers to iterate on prompts in a playground-style experience and access new model settings to optimize Claude’s behavior. The user can create multiple prompts and navigate between them for different projects, and revisions are saved as they go to retain historical context. Developers can also generate code snippets to use their prompts directly in one of our SDKs. Access to the console is by invitation only based on when this content was published.

Anthropic will empower developers by adding system prompts, allowing users to provide custom instructions to Claude to improve performance. System prompts set helpful context that enhances Claude’s ability to assume specified personalities and roles or structure responses in a more customizable, consistent way that aligns with user needs.

Claude 2.1 is available in their API and powers the chat interface at claude.ai for both the free and Pro tiers. This advantage is for those who want to test drive before committing to Pro. Usage of the 200K token context window is reserved for Claude Pro users, who can now upload larger files.

Overall, I am happy to see these improvements with Claude 2.1. I like having choices in this space and more opportunities to learn about LLM in AI as a technology person interested in large data sets. Claude is on my shortlist.

Originally published at https://mstechdiva.com on November 23, 2023.

Open Source AI Gets the Bird

Open source creates opportunities for developers worldwide to work together on projects, share knowledge and collectively enhance software solutions. This inclusive approach not speeds up advancements but also ensures that cutting edge tools and technologies are available to everyone. So it always warms my heart when I see any innovations in this space.

Open source software drives innovation by reducing development costs and ensuring transparency and security. To me it embodies the essence of intelligence, by bringing developers together to learn from each other and shape the future of technology as a united community.

The artificial intelligence community has reached a significant milestone with the introduction of Falcon 180B, an open-source large language model (LLM) that boasts an astonishing 180 billion parameters, trained on an unprecedented volume of data. This groundbreaking release, announced by the Hugging Face AI community in a recent blog post, has already profoundly impacted the field. Falcon 180B builds upon the success of its predecessors in the Falcon series, introducing innovations such as multi-query attention to achieve its impressive scale, trained on a staggering 3.5 trillion tokens, representing the longest single-epoch pretraining for any open-source model to date.

Scaling Unleashed

Achieving this goal was no small endeavor. Falcon 180B required the coordinated power of 4,096 GPUs working simultaneously for approximately 7 million GPU hours, with the training and refinement process orchestrated through Amazon SageMaker. Considering this regarding the size of the LLM, the model’s parameters measure 2.5 times larger than Meta’s LLaMA 2, which had previously been considered the most capable open-source LLM with 70 billion parameters trained on 2 trillion tokens. The numbers and data involved are staggering, its like an analyst dream.

Performance Breakthrough

Falcon 180B isn’t just about scale; it excels in benchmark performance across various natural language processing (NLP) tasks. On the leaderboard for open-access models, it impressively scores 68.74 points, coming close to commercial giants like Google’s PaLM-2 on the HellaSwag benchmark. It matches or exceeds PaLM-2 Medium on commonly used benchmarks like HellaSwag, LAMBADA, WebQuestions, Winogrande, and more and performs on par with Google’s PaLM-2 Large. This level of performance is a testament to the capabilities of open-source models, even when compared to industry giants.

Comparing with ChatGPT

When measured against ChatGPT, Falcon 180B sits comfortably between GPT 3.5 and GPT4, depending on the evaluation benchmark. While it may not surpass the capabilities of the paid “plus” version of ChatGPT, it certainly gives the free version a run. I am always happy to see this type of healthy competition in this space.

The Huggingface community is strong so there is potential for further fine-tuning by the community, which is expected to yield even more impressive results. Falcon 180 B’s open release marks a significant step forward in the rapid evolution of large language models, showcasing advanced natural language processing capabilities right from the outset.

A New Chapter in Efficiency

Beyond its sheer scale, Falcon 180B embodies the progress in training large AI models more efficiently. Techniques such as LoRAs, weight randomization, and Nvidia’s Perfusion have played pivotal roles in achieving this efficiency, heralding a new era in AI model development.

With Falcon 180B now freely available on Hugging Face, the AI research community eagerly anticipates further enhancements and refinements. This release marks a huge advancement for open-source AI, setting the stage for exciting developments and breakthroughs. Falcon 180B has already demonstrated its potential to redefine the boundaries of what’s possible in the world of artificial intelligence, and its journey is just beginning. It’s the numbers for me. I am always happy to see this growth in this space. Yes, “the bird” was always about technology. Shared references give you a great headstart in understanding all about Falcon.

References:

huggingface on GitHub

huggingface Falcon documentation

Falcon Models from Technlogy Innovation Institute