progressEdd's llms, ai, and dev corner

progressEdd · July 24, 2024, 4:14am

Hi I’m progressEdd, I build AI and AI Accessories.

I wanted to formally start my own corner for exploring open source models and anything AI related. Stick around if you are interested.

SgtAwesomesauce · September 14, 2024, 1:47am

2 posts were merged into an existing topic: Novasty’s Scratch Thread

progressEdd · August 3, 2024, 11:40pm

With the release of llama 3.1, I decided to revisit this code. I learned the availability of new models through the ollama and lmstudio discord servers

Initially I thought the issue was with the system prompt of lmstudio, but when I tried ollama with the same model

github.com

slabstech/llm-recipes/blob/main/tutorials/autogen/notebook/mixtral_agentchat_planning.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/agentchat_planning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Auto Generated Agent Chat: Collaborative Task Solving with Coding and Planning Agent\n",

This file has been truncated. show original

The outputs between the two models were the same.

I was going to post a thread as a help/support thread within the autogen discord, but it wouldn’t be accessible to the public it’s hard to find stuff on discord servers.

github.com/microsoft/autogen

[Question][Issue]: Open Source Models Ignore Tools

opened 12:13AM - 24 Jul 24 UTC

progressEdd

function/tool alt-models

### Describe the issue Hi all I'm trying to replicate the code in this tutorial…, with a local llm using LM Studio endpoint https://microsoft.github.io/autogen/docs/tutorial/tool-use When I initiate the chat, open source models ignore the tools and instead opt to solve the calculate the result using order of operations. I am able to replicate it using the AzureOpenAi GPT4 API. ### Steps to reproduce # pure python we expect the answer to be `221492.8` ```python (44232 + 13312 / (232 - 32)) * 5 ``` # llama 3.1 ## code I ran this code for llama 3.1 ```python ollama31 = { "config_list": [ { "model": "lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF", "base_url": "http://localhost:1234/v1", "api_key": "lm-studio", }, ], "cache_seed": None, # Disable caching. } import os from autogen import ConversableAgent # Let's first define the assistant agent that suggests tool calls using gemma assistant = ConversableAgent( name="Assistant", system_message="You are a helpful AI assistant. " "You can help with simple calculations." "Return 'TERMINATE' when the task is done.", llm_config=ollama31, ) # The user proxy agent is used for interacting with the assistant agent # and executes tool calls. user_proxy = ConversableAgent( name="User", llm_config=False, is_termination_msg=lambda msg: msg.get("content") is not None and "TERMINATE" in msg["content"], human_input_mode="NEVER", ) from typing import Annotated, Literal Operator = Literal["+", "-", "*", "/"] def calculator(a: int, b: int, operator: Annotated[Operator, "operator"]) -> int: print("Calculating", a, operator, b) if operator == "+": return a + b elif operator == "-": return a - b elif operator == "*": return a * b elif operator == "/": return int(a / b) else: raise ValueError("Invalid operator") # Register the tool signature with the assistant agent. assistant.register_for_llm(name="calculator", description="A simple calculator")(calculator) # Register the tool function with the user proxy agent. user_proxy.register_for_execution(name="calculator")(calculator) from autogen import register_function # Register the calculator function to the two agents. register_function( calculator, caller=assistant, # The assistant agent can suggest calls to the calculator. executor=user_proxy, # The user proxy agent can execute the calculator calls. name="calculator", # By default, the function name is used as the tool name. description="A simple calculator", # A description of the tool. ) import autogen from autogen import AssistantAgent, UserProxyAgent # Start logging using the agentchat_logging notebook logging_session_id = autogen.runtime_logging.start(config={"dbname": "logs.db"}) print("Logging session ID: " + str(logging_session_id)) chat_result = user_proxy.initiate_chat(assistant, message="What is (44232 + 13312 / (232 - 32)) * 5?") autogen.runtime_logging.stop() ``` ## output it seemed to hallucinate in the add `(44232 + 13312)` and subtract `(44232 + 66.56)` operations ```python Logging session ID: 2631a5b5-ff5c-4b95-8e30-33b8af4269c5 User (to Assistant): What is (44232 + 13312 [/](https://file+.vscode-resource.vscode-cdn.net/) (232 - 32)) * 5? -------------------------------------------------------------------------------- >>>>>>>> USING AUTO REPLY... [autogen.oai.client: 07-23 18:08:46] {329} WARNING - Model lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing. Assistant (to User): To evaluate this expression, we need to follow the order of operations (PEMDAS): 1. Subtract 32 from 232: 232 - 32 = 200 2. Divide 13312 by 200: 13312 [/](https://file+.vscode-resource.vscode-cdn.net/) 200 = 66.56 3. Add 44232 and 66.56: 44232 + 66.56 = 44398.56 4. Multiply the result by 5: 44398.56 * 5 = 221992.8 Therefore, (44232 + 13312 [/](https://file+.vscode-resource.vscode-cdn.net/) (232 - 32)) * 5 = 221992.8 -------------------------------------------------------------------------------- User (to Assistant): -------------------------------------------------------------------------------- >>>>>>>> USING AUTO REPLY... ``` # gemma 2 weirdly enough gemma didn't have issues doing the calculation. ## code ```python gemma = { "config_list": [ { "model": "lmstudio-community/gemma-2-9b-it-GGUF", "base_url": "http://localhost:1234/v1", "api_key": "lm-studio", }, ], "cache_seed": None, # Disable caching. } import os from autogen import ConversableAgent # Let's first define the assistant agent that suggests tool calls using gemma assistant = ConversableAgent( name="Assistant", system_message="You are a helpful AI assistant. " "You can help with simple calculations. " "Return 'TERMINATE' when the task is done.", llm_config=gemma, ) # The user proxy agent is used for interacting with the assistant agent # and executes tool calls. user_proxy = ConversableAgent( name="User", llm_config=False, is_termination_msg=lambda msg: msg.get("content") is not None and "TERMINATE" in msg["content"], human_input_mode="NEVER", ) # Register the tool signature with the assistant agent. assistant.register_for_llm(name="calculator", description="A simple calculator")(calculator) # Register the tool function with the user proxy agent. user_proxy.register_for_execution(name="calculator")(calculator) from typing import Annotated, Literal Operator = Literal["+", "-", "*", "/"] def calculator(a: int, b: int, operator: Annotated[Operator, "operator"]) -> int: print("Calculating", a, operator, b) if operator == "+": return a + b elif operator == "-": return a - b elif operator == "*": return a * b elif operator == "/": return int(a / b) else: raise ValueError("Invalid operator") from autogen import register_function # Register the calculator function to the two agents. register_function( calculator, caller=assistant, # The assistant agent can suggest calls to the calculator. executor=user_proxy, # The user proxy agent can execute the calculator calls. name="calculator", # By default, the function name is used as the tool name. description="A simple calculator", # A description of the tool. ) import autogen from autogen import AssistantAgent, UserProxyAgent # Start logging using the agentchat_logging notebook logging_session_id = autogen.runtime_logging.start(config={"dbname": "logs.db"}) print("Logging session ID: " + str(logging_session_id)) chat_result = user_proxy.initiate_chat(assistant, message="What is (44232 + 13312 / (232 - 32)) * 5?") autogen.runtime_logging.stop() ``` ## output ```python Logging session ID: e170cbf8-a707-44e1-a24b-63f985455f81 User (to Assistant): What is (44232 + 13312 [/](https://file+.vscode-resource.vscode-cdn.net/) (232 - 32)) * 5? -------------------------------------------------------------------------------- >>>>>>>> USING AUTO REPLY... [autogen.oai.client: 07-16 15:24:35] {329} WARNING - Model lmstudio-community/gemma-2-9b-it-GGUF/gemma-2-9b-it-Q6_K.gguf is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing. Assistant (to User): Here's how to solve that calculation: 1. **Parentheses first:** 232 - 32 = 200 2. **Division:** 13312 [/](https://file+.vscode-resource.vscode-cdn.net/) 200 = 66.56 3. **Addition:** 44232 + 66.56 = 44298.56 4. **Multiplication:** 44298.56 * 5 = 221492.8 **Answer:** 221492.8 TERMINATE ``` ### Screenshots and logs # lm studio config ## llama3 ```json { "name": "Exported from LM Studio on 7/23/2024, 7:07:38 PM", "load_params": { "n_ctx": 4000, "n_batch": 512, "rope_freq_base": 0, "rope_freq_scale": 0, "n_gpu_layers": 10, "use_mlock": true, "main_gpu": 0, "tensor_split": [ 0 ], "seed": -1, "f16_kv": true, "use_mmap": true, "no_kv_offload": false, "num_experts_used": 0 }, "inference_params": { "n_threads": 4, "n_predict": -1, "top_k": 40, "min_p": 0.05, "top_p": 0.95, "temp": 0.8, "repeat_penalty": 1.1, "input_prefix": "<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n", "input_suffix": "<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n", "antiprompt": [ "<|start_header_id|>", "<|eot_id|>" ], "pre_prompt": "You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.", "pre_prompt_suffix": "", "pre_prompt_prefix": "<|start_header_id|>system<|end_header_id|>\\n\\n", "seed": -1, "tfs_z": 1, "typical_p": 1, "repeat_last_n": 64, "frequency_penalty": 0, "presence_penalty": 0, "n_keep": 0, "logit_bias": {}, "mirostat": 0, "mirostat_tau": 5, "mirostat_eta": 0.1, "memory_f16": true, "multiline_input": false, "penalize_nl": true } } ``` ## gemma 2 config ```json { "name": "Exported from LM Studio on 7/23/2024, 7:09:40 PM", "load_params": { "n_ctx": 4000, "n_batch": 512, "rope_freq_base": 0, "rope_freq_scale": 0, "n_gpu_layers": 10, "use_mlock": true, "main_gpu": 0, "tensor_split": [ 0 ], "seed": -1, "f16_kv": true, "use_mmap": true, "no_kv_offload": false, "num_experts_used": 0 }, "inference_params": { "n_threads": 4, "n_predict": -1, "top_k": 40, "min_p": 0.05, "top_p": 0.95, "temp": 0.8, "repeat_penalty": 1.1, "input_prefix": "<start_of_turn>user\\n", "input_suffix": "<end_of_turn>\\n<start_of_turn>model\\n", "antiprompt": [ "<start_of_turn>user", "<start_of_turn>model", "<end_of_turn>" ], "pre_prompt": "", "pre_prompt_suffix": "", "pre_prompt_prefix": "", "seed": -1, "tfs_z": 1, "typical_p": 1, "repeat_last_n": 64, "frequency_penalty": 0, "presence_penalty": 0, "n_keep": 0, "logit_bias": {}, "mirostat": 0, "mirostat_tau": 5, "mirostat_eta": 0.1, "memory_f16": true, "multiline_input": false, "penalize_nl": true } } ``` ### Additional Information poetry pyproject dependencies ``` toml [tool.poetry.dependencies] python = ">=3.10,<3.13" pandas = "*" streamlit = "^1.31.0" langchain = "^0.1.5" python-dotenv = "^1.0.1" emoji = "^2.10.1" async-timeout = "^4.0.3" numpy = "*" Pillow = "*" openai = "*" ipykernel = "^6.29.2" scikit-learn = "^1.4.0" ipywidgets = "^8.1.2" openpyxl = "*" kaleido = "0.2.1" langchain-community = "^0.0.21" langchain-openai = "^0.0.6" faker = "^24.0.0" langchain-experimental = "^0.0.52" tabulate = "^0.9.0" geopy = "^2.4.1" panel = "^1.3.8" jupyter-bokeh = "^4.0.1" pyviz-comms = "^3.0.2" nltk = "^3.8.1" lxml = "^5.2.1" fuzzywuzzy = "^0.18.0" python-levenshtein = "^0.25.1" googlemaps = "^4.10.0" pyautogen = {extras = ["long-context"], version = "^0.2.31"} datasets = "^2.20.0" ```

macdos359 · September 13, 2024, 12:11pm

I am a computer janitor getting into coding, and lordy this stuff has captured my interest. I have a locally hosted Llama3.1:8b running in a box, and I am learning how to configure a RAG. I will be around to see how you are doing here! Nice to meet you!

ubergarm · September 13, 2024, 4:20pm

LLMs and Image Diffusion models are definitely fun to play with locally to push the limits of your build!

I’m running on Linux as there are a variety of inference engines available. Typically I use good old llama.cpp by git pulling it and doing build. (Building llama.cpp with make -j($nproc) was a reliable way to crash my old degraded i9-14900k btw hah…)

The biggest pro of llama.cpp is support for partial offload of GGUF models. With 96GB RAM and 24GB VRAM I can run Llama-3.1-70B IQ3_XXS at just over 7 tok/sec (maybe 8k kv cache fp16). I tried litellm but wanted llama.cpp specific support so made my own python async library for streaming llama-cpp-api-client. Not perfect, but was able to build a simple discord bot with it and kick the tires on some models.

I may try Mistral-Large or Deepseek-v2 / v2.5 in the biggest quant that will memory map into 96GB soon. It is unfortunate that I’m limited to 2x DIMMs to get full memory bandwidth, as these big LLMs gobble up RAM fast when inferencing on CPU but are so bottle-necked by bandwidth. Will be lucky to get 2 tok/sec with the big models probably hah…

If you have enough VRAM to avoid offload, (or use 8B models), you can get faster inference with other engines. Pretty good recent benchmark on r/localllama the other day.

For Image Diffusion I’m using ComfyUI and mostly FLUX.1-dev with random LoRA’s from civit haha… It is kind of a wild space though, so be prepared to sort through a lot of “interesting” models lol…

Generally I run my 3090TI FE cap’d at 300 watts and have not yet added a second GPU to the mix as tbh the quality isn’t there yet imho…

Cheers, curious to hear how your adventures go!

progressEdd · July 25, 2024, 12:49am

got the suggestion to use litellm as a model endpoint working. TIL litellm as a simple open source model endpoint. The performance isn’t as good, but I suspect the ollama models don’t have the same quantization as LM studio. LM studio offers Q3-Q8 levels of quantization the lower a model is quantized the more constrained it is.
For those who want to read more see this article

What is Quantization and Why is it Important?

Quantization is a model compression technique that converts the weights and activations within an LLM from a high-precision data representation to a lower-precision data representation, i.e., from a data type that can hold more information to one that holds less. A typical example of this is the conversion of data from a 32-bit floating-point number (FP32) to an 8-bit or 4-bit integer (INT4 or INT8).

A great analogy for understanding quantization is image compression. Compressing an image involves reducing its size by removing some of the information, i.e., bits of data, from it. Now, while decreasing the size of an image typically reduces its quality (to acceptable levels), it also means more images can be saved on a given device while requiring less time and bandwidth to transfer or display to a user. In a similar way, quantizing an LLM increases its portability and the number of ways it can be deployed – albeit with an acceptable sacrifice to detail or precision.

Quantization is an important process within machine learning because reducing the number of bits required for each of a model’s weights adds up to a significant decrease in its overall size. Consequently, quantization produces LLMs that consume less memory, require less storage space, are more energy-efficient, and are capable of faster inference. This all adds up to the critical advantage of enabling LLMs to run on a wider range of devices, including single GPUs, instead of expensive hardware featuring multiple GPUs, and, in some cases, even CPUs.

github.com/microsoft/autogen

[Question][Issue]: Open Source Models Ignore Tools

opened 12:13AM - 24 Jul 24 UTC

progressEdd

function/tool alt-models

### Describe the issue Hi all I'm trying to replicate the code in this tutorial…, with a local llm using LM Studio endpoint https://microsoft.github.io/autogen/docs/tutorial/tool-use When I initiate the chat, open source models ignore the tools and instead opt to solve the calculate the result using order of operations. I am able to replicate it using the AzureOpenAi GPT4 API. ### Steps to reproduce # pure python we expect the answer to be `221492.8` ```python (44232 + 13312 / (232 - 32)) * 5 ``` # llama 3.1 ## code I ran this code for llama 3.1 ```python ollama31 = { "config_list": [ { "model": "lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF", "base_url": "http://localhost:1234/v1", "api_key": "lm-studio", }, ], "cache_seed": None, # Disable caching. } import os from autogen import ConversableAgent # Let's first define the assistant agent that suggests tool calls using gemma assistant = ConversableAgent( name="Assistant", system_message="You are a helpful AI assistant. " "You can help with simple calculations." "Return 'TERMINATE' when the task is done.", llm_config=ollama31, ) # The user proxy agent is used for interacting with the assistant agent # and executes tool calls. user_proxy = ConversableAgent( name="User", llm_config=False, is_termination_msg=lambda msg: msg.get("content") is not None and "TERMINATE" in msg["content"], human_input_mode="NEVER", ) from typing import Annotated, Literal Operator = Literal["+", "-", "*", "/"] def calculator(a: int, b: int, operator: Annotated[Operator, "operator"]) -> int: print("Calculating", a, operator, b) if operator == "+": return a + b elif operator == "-": return a - b elif operator == "*": return a * b elif operator == "/": return int(a / b) else: raise ValueError("Invalid operator") # Register the tool signature with the assistant agent. assistant.register_for_llm(name="calculator", description="A simple calculator")(calculator) # Register the tool function with the user proxy agent. user_proxy.register_for_execution(name="calculator")(calculator) from autogen import register_function # Register the calculator function to the two agents. register_function( calculator, caller=assistant, # The assistant agent can suggest calls to the calculator. executor=user_proxy, # The user proxy agent can execute the calculator calls. name="calculator", # By default, the function name is used as the tool name. description="A simple calculator", # A description of the tool. ) import autogen from autogen import AssistantAgent, UserProxyAgent # Start logging using the agentchat_logging notebook logging_session_id = autogen.runtime_logging.start(config={"dbname": "logs.db"}) print("Logging session ID: " + str(logging_session_id)) chat_result = user_proxy.initiate_chat(assistant, message="What is (44232 + 13312 / (232 - 32)) * 5?") autogen.runtime_logging.stop() ``` ## output it seemed to hallucinate in the add `(44232 + 13312)` and subtract `(44232 + 66.56)` operations ```python Logging session ID: 2631a5b5-ff5c-4b95-8e30-33b8af4269c5 User (to Assistant): What is (44232 + 13312 [/](https://file+.vscode-resource.vscode-cdn.net/) (232 - 32)) * 5? -------------------------------------------------------------------------------- >>>>>>>> USING AUTO REPLY... [autogen.oai.client: 07-23 18:08:46] {329} WARNING - Model lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing. Assistant (to User): To evaluate this expression, we need to follow the order of operations (PEMDAS): 1. Subtract 32 from 232: 232 - 32 = 200 2. Divide 13312 by 200: 13312 [/](https://file+.vscode-resource.vscode-cdn.net/) 200 = 66.56 3. Add 44232 and 66.56: 44232 + 66.56 = 44398.56 4. Multiply the result by 5: 44398.56 * 5 = 221992.8 Therefore, (44232 + 13312 [/](https://file+.vscode-resource.vscode-cdn.net/) (232 - 32)) * 5 = 221992.8 -------------------------------------------------------------------------------- User (to Assistant): -------------------------------------------------------------------------------- >>>>>>>> USING AUTO REPLY... ``` # gemma 2 weirdly enough gemma didn't have issues doing the calculation. ## code ```python gemma = { "config_list": [ { "model": "lmstudio-community/gemma-2-9b-it-GGUF", "base_url": "http://localhost:1234/v1", "api_key": "lm-studio", }, ], "cache_seed": None, # Disable caching. } import os from autogen import ConversableAgent # Let's first define the assistant agent that suggests tool calls using gemma assistant = ConversableAgent( name="Assistant", system_message="You are a helpful AI assistant. " "You can help with simple calculations. " "Return 'TERMINATE' when the task is done.", llm_config=gemma, ) # The user proxy agent is used for interacting with the assistant agent # and executes tool calls. user_proxy = ConversableAgent( name="User", llm_config=False, is_termination_msg=lambda msg: msg.get("content") is not None and "TERMINATE" in msg["content"], human_input_mode="NEVER", ) # Register the tool signature with the assistant agent. assistant.register_for_llm(name="calculator", description="A simple calculator")(calculator) # Register the tool function with the user proxy agent. user_proxy.register_for_execution(name="calculator")(calculator) from typing import Annotated, Literal Operator = Literal["+", "-", "*", "/"] def calculator(a: int, b: int, operator: Annotated[Operator, "operator"]) -> int: print("Calculating", a, operator, b) if operator == "+": return a + b elif operator == "-": return a - b elif operator == "*": return a * b elif operator == "/": return int(a / b) else: raise ValueError("Invalid operator") from autogen import register_function # Register the calculator function to the two agents. register_function( calculator, caller=assistant, # The assistant agent can suggest calls to the calculator. executor=user_proxy, # The user proxy agent can execute the calculator calls. name="calculator", # By default, the function name is used as the tool name. description="A simple calculator", # A description of the tool. ) import autogen from autogen import AssistantAgent, UserProxyAgent # Start logging using the agentchat_logging notebook logging_session_id = autogen.runtime_logging.start(config={"dbname": "logs.db"}) print("Logging session ID: " + str(logging_session_id)) chat_result = user_proxy.initiate_chat(assistant, message="What is (44232 + 13312 / (232 - 32)) * 5?") autogen.runtime_logging.stop() ``` ## output ```python Logging session ID: e170cbf8-a707-44e1-a24b-63f985455f81 User (to Assistant): What is (44232 + 13312 [/](https://file+.vscode-resource.vscode-cdn.net/) (232 - 32)) * 5? -------------------------------------------------------------------------------- >>>>>>>> USING AUTO REPLY... [autogen.oai.client: 07-16 15:24:35] {329} WARNING - Model lmstudio-community/gemma-2-9b-it-GGUF/gemma-2-9b-it-Q6_K.gguf is not found. The cost will be 0. In your config_list, add field {"price" : [prompt_price_per_1k, completion_token_price_per_1k]} for customized pricing. Assistant (to User): Here's how to solve that calculation: 1. **Parentheses first:** 232 - 32 = 200 2. **Division:** 13312 [/](https://file+.vscode-resource.vscode-cdn.net/) 200 = 66.56 3. **Addition:** 44232 + 66.56 = 44298.56 4. **Multiplication:** 44298.56 * 5 = 221492.8 **Answer:** 221492.8 TERMINATE ``` ### Screenshots and logs # lm studio config ## llama3 ```json { "name": "Exported from LM Studio on 7/23/2024, 7:07:38 PM", "load_params": { "n_ctx": 4000, "n_batch": 512, "rope_freq_base": 0, "rope_freq_scale": 0, "n_gpu_layers": 10, "use_mlock": true, "main_gpu": 0, "tensor_split": [ 0 ], "seed": -1, "f16_kv": true, "use_mmap": true, "no_kv_offload": false, "num_experts_used": 0 }, "inference_params": { "n_threads": 4, "n_predict": -1, "top_k": 40, "min_p": 0.05, "top_p": 0.95, "temp": 0.8, "repeat_penalty": 1.1, "input_prefix": "<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n", "input_suffix": "<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n", "antiprompt": [ "<|start_header_id|>", "<|eot_id|>" ], "pre_prompt": "You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.", "pre_prompt_suffix": "", "pre_prompt_prefix": "<|start_header_id|>system<|end_header_id|>\\n\\n", "seed": -1, "tfs_z": 1, "typical_p": 1, "repeat_last_n": 64, "frequency_penalty": 0, "presence_penalty": 0, "n_keep": 0, "logit_bias": {}, "mirostat": 0, "mirostat_tau": 5, "mirostat_eta": 0.1, "memory_f16": true, "multiline_input": false, "penalize_nl": true } } ``` ## gemma 2 config ```json { "name": "Exported from LM Studio on 7/23/2024, 7:09:40 PM", "load_params": { "n_ctx": 4000, "n_batch": 512, "rope_freq_base": 0, "rope_freq_scale": 0, "n_gpu_layers": 10, "use_mlock": true, "main_gpu": 0, "tensor_split": [ 0 ], "seed": -1, "f16_kv": true, "use_mmap": true, "no_kv_offload": false, "num_experts_used": 0 }, "inference_params": { "n_threads": 4, "n_predict": -1, "top_k": 40, "min_p": 0.05, "top_p": 0.95, "temp": 0.8, "repeat_penalty": 1.1, "input_prefix": "<start_of_turn>user\\n", "input_suffix": "<end_of_turn>\\n<start_of_turn>model\\n", "antiprompt": [ "<start_of_turn>user", "<start_of_turn>model", "<end_of_turn>" ], "pre_prompt": "", "pre_prompt_suffix": "", "pre_prompt_prefix": "", "seed": -1, "tfs_z": 1, "typical_p": 1, "repeat_last_n": 64, "frequency_penalty": 0, "presence_penalty": 0, "n_keep": 0, "logit_bias": {}, "mirostat": 0, "mirostat_tau": 5, "mirostat_eta": 0.1, "memory_f16": true, "multiline_input": false, "penalize_nl": true } } ``` ### Additional Information poetry pyproject dependencies ``` toml [tool.poetry.dependencies] python = ">=3.10,<3.13" pandas = "*" streamlit = "^1.31.0" langchain = "^0.1.5" python-dotenv = "^1.0.1" emoji = "^2.10.1" async-timeout = "^4.0.3" numpy = "*" Pillow = "*" openai = "*" ipykernel = "^6.29.2" scikit-learn = "^1.4.0" ipywidgets = "^8.1.2" openpyxl = "*" kaleido = "0.2.1" langchain-community = "^0.0.21" langchain-openai = "^0.0.6" faker = "^24.0.0" langchain-experimental = "^0.0.52" tabulate = "^0.9.0" geopy = "^2.4.1" panel = "^1.3.8" jupyter-bokeh = "^4.0.1" pyviz-comms = "^3.0.2" nltk = "^3.8.1" lxml = "^5.2.1" fuzzywuzzy = "^0.18.0" python-levenshtein = "^0.25.1" googlemaps = "^4.10.0" pyautogen = {extras = ["long-context"], version = "^0.2.31"} datasets = "^2.20.0" ```

progressEdd · November 13, 2024, 10:45pm

There has been a lot of headlines discussing the diminishing returns of throwing more data, model parameters, and compute to improve llm performance. This is a great math/technical explanation why performance gains is slowing

I got lost around 20:01 when the video had the visual explanation behind dataset scaling

progressEdd · November 18, 2024, 10:20pm

The meme

made this meme of Dario Amodei and Yann LeCun

Also posted on twitter too
https://x.com/ProgressEdd/status/1858634573942337998

Chatgpt helped with the phrasing and making dario transparent

How did this meme spawn?

The edit spawned during some messages with some of my ai friends going over the following article about llms for chip design. I noticed the following phrase

my friend responded with this quote

after realizing the duality of man, I knew I had to create the I have 2 sides meme but with the AI Leaders Dario Amodei and Yann LeCun who have semi opposing takes on AI and AGI

Darin755 · November 18, 2024, 10:21pm

ollama + Mistral or Mixtral

progressEdd · November 18, 2024, 10:35pm

Maybe for brainstorming phrasing, have you found any good front ends for image instructions? LMStudio is definitely up there, but not as convenient as chatgpt.

I’ll write a post for the making of this meme

Darin755 · November 19, 2024, 12:04am

Alpaca or OpenWebUI

progressEdd · November 19, 2024, 5:17am

will have to look into it

progressEdd · November 22, 2024, 4:52am

While replying to this post, I remembered my conversation with a Intel rep at microsoft ignite about model quantization and how it scales to hardware, as well as his experience using with quantized image models with flux.

Apparently with model quantization, you can change the weight data types for each layer within the neural network.

In a unquantized model, the weights are stored as floating point 32 bit integer (fp32) datatype, and as we quantize the model, we reduce the fp32 bits down to lower levels for example into 8. He showed me that as the quantization is lowered, the quality of outputs decreases, but the latency is much faster

If that explanation didn’t make sense, this hugging face article goes more in depth into the techniques to

I may butcher the explanation, but with hardware, having more memory allows for more weights to be stored, and having dedicated neural processing units reduces the amount of latency to process the weight calculations.

igormp · November 22, 2024, 5:58pm

That depends on hardware support, fwiw.

FP16 has great support in modern-ish hardware and gets pretty much double the speed of your regular fp32 weights model, with basically no performance loss.

Going down to smaller data types (be int or float), will save you on memory since the data is smaller (duh), but the performance depends on the underlying hardware support. For CPU use, that’s pretty much irrelevant.
But for GPUs or other dedicated hardware, it may not support the specific format, and thus performance won’t be sped up and will be slower than your regular fp32/16 inference run.

Of course the above is moot if we’re talking a model that’s normally way too big for your memory, since running a quantized model, no matter the speed, is going to be way faster than not being able to run at all lol

progressEdd · November 22, 2024, 7:41pm

like my hardware M2 pro or a fedora 5800x3d + 2070 8gb?

neat I didn’t know about that

Based on this, I’m assuming you mean latency when you are talking about performance in the previous sentence.

yeah I’m really interested to see how far companies can push small models. Microsoft’s Phi and Apple seem to be leading the charge on this front

igormp · November 22, 2024, 8:24pm

Modern as in your GPU (Turing) has tensor cores with support for mixed precision. FP16 inference should be twice as fast than regular FP32.

No idea on the specifics of the M2 chips, nor the software support for those.

I assume that by latency they meant the time it takes to perform a single feed-forward step, so yeah, more performance = less latency.

I mean, you can have a 10B Q8 model, and a 20B Q4 model. The former has half the parameters, but is as when it comes to memory to the latter.

I see research advancing in both having larger models with less precise weights (like those 1-bit models) and models with less parameters (like the smaller Phi models from MS), which is indeed very nice.

progressEdd · December 9, 2024, 1:32am

This was a really interesting read that I want to buy the book

progressEdd · December 13, 2024, 2:37pm

Had a colleague ask me about image editing models. This was a pretty cool blog

progressEdd · January 11, 2025, 5:19am

This video inspired me to go back and make some progress some of my past projects
For example I updated my python environment setup guide

The old poetry guide from

github.com

progressEdd/dev-onboarding/blob/master/python-environments/python-virtual-environments.md#installing-poetry-following-the-developers-best-practices

# Setting up python environments
This guide assumes that VSCode/VSCodium has been installed to your system

## Installing UV
1. Navigate to the following [Getting started page](https://docs.astral.sh/uv/#getting-started)
2. Run the installer script for your platform
3. Initialize or build your project
   - If you are working on a new project, run `uv init $project_name`
     - replace `$project_name` with a directory name
   - If you have been given a project with a `pyproject.toml`, run `uv sync` within the same directory as the `pyproject.toml` file
4. If vscode/jupyter python interpreter to the`.venv` created by uv
    - When using notebooks make sure to select the local .venv for your python interpreter
       1. ![](../supporting_files/images/python-virtual-environments/edits/20240626152533.png)
       2. ![](../supporting_files/images/python-virtual-environments/edits/20240626152602.png)
       3. ![](../supporting_files/images/python-virtual-environments/edits/20240626152621.png)
5. Install or declare your UV dependencies
    - instead of `pip install $package_name` you run `uv add $package_name`. Use the `uv add` when possible as it will automatically update the `pyproject.toml`
    - a sample file has been provided in [`pyproject.toml`](../python-environments/pyproject.toml) 

## Installing poetry following the developer's best practices

This file has been truncated. show original

is this meme

progressEdd · January 24, 2025, 8:19pm

ag2 annoucement after the microsoft split
https://fixupx.com/ag2oss/status/1882878967713259705