PriceBench

Complete code files for data curation and training can be found in the GitHub Repository.

📊 Data Curation Process

1. Dataset Selection and Initial Loading

The training process began with the Amazon Reviews 2023 dataset from Hugging Face, specifically focusing on multiple product categories including:

• Automotive
• Electronics
• Office Products
• Tools and Home Improvement
• Cell Phones and Accessories
• Toys and Games
• Appliances
• Musical Instruments

2. Data Filtering and Quality Control

A rigorous filtering process was implemented to ensure high-quality training data:

• Price Range: Only products with prices between $0.50 and $999.49 were included
• Content Requirements: Products needed minimum 300 characters of description content
• Token Limits: Text was truncated to 150-160 tokens to maintain consistency
• Data Cleaning: Removal of irrelevant metadata, product codes, and standardization of text formats

3. Text Processing and Tokenization

The Item class handled sophisticated text preprocessing:

• Combination of product title, description, features, and technical details
• Removal of unnecessary characters, brackets, and product-specific codes
• Tokenization using Meta-Llama-3.1-8B tokenizer for consistency with the base model
• Smart truncation to maintain semantic meaning within token limits

4. Dataset Balancing and Sampling

To create a well-balanced training dataset:

• Price Distribution: Implemented weighted sampling to reduce bias toward low-priced items
• Category Balance: Applied category-specific weights to prevent over-representation of certain product types
• Final Dataset: Curated 402,000 high-quality product entries with balanced price distribution (average ~$125)

5. Prompt Engineering

Each training example was formatted with a consistent structure:

• Question: "How much does this cost to the nearest dollar?"
• Product Information: Cleaned and tokenized product details
• Answer Format: "Price is $XXX.00" for consistent completion learning

🚀 Model Training Process

6. Base Model Selection

Meta-Llama-3.1-8B was chosen as the foundation model due to its:

• Excellent numerical reasoning capabilities
• Efficient tokenization of numbers (1-999 map to single tokens)
• Strong instruction-following abilities
• Optimal balance between performance and computational requirements

7. Quantization and Memory Optimization

To enable efficient training on consumer hardware:

• 4-bit Quantization: Used BitsAndBytesConfig with NF4 quantization
• Double Quantization: Applied additional compression for memory efficiency
• Memory Footprint: Reduced model size to approximately 4.5GB

8. QLoRA Fine-tuning Configuration

Parameter-efficient fine-tuning was implemented using QLoRA:

• LoRA Rank (r): 32 for optimal parameter efficiency
• LoRA Alpha: 64 for scaling factor
• Target Modules: Applied to attention layers (q_proj, k_proj, v_proj, o_proj)
• Dropout: 0.1 to prevent overfitting

9. Training Hyperparameters

Carefully tuned training configuration:

• Learning Rate: 1e-4 with cosine scheduling
• Batch Size: 1 with gradient accumulation
• Epochs: 1 (sufficient due to large dataset size)
• Optimizer: Paged AdamW 32-bit for memory efficiency
• Sequence Length: 182 tokens maximum

10. Specialized Data Collation

Used DataCollatorForCompletionOnlyLM to:

• Focus loss calculation only on price predictions
• Ignore input tokens during loss computation
• Enable efficient completion-based learning

11. Training Infrastructure

Professional training setup included:

• Platform: Google Colab with GPU acceleration
• Monitoring: Weights & Biases integration for experiment tracking
• Version Control: Automatic model versioning and Hub integration
• Checkpointing: Regular saves every 50 steps for reliability

12. Model Evaluation and Testing

Comprehensive evaluation framework with custom Tester class:

• Error Metrics: Mean Absolute Error (MAE) and Root Mean Square Logarithmic Error (RMSLE)
• Color-coded Results: Green (excellent), Orange (good), Red (needs improvement)
• Scatter Plot Analysis: Visual comparison between predictions and ground truth
• Hit Rate Calculation: Percentage of predictions within acceptable error margins

The result is a highly specialized pricing model that achieves professional-grade accuracy while maintaining computational efficiency and cost-effectiveness.

🏆 Performance Benchmarking Results

PriceBench was evaluated against a comprehensive range of models, from traditional machine learning approaches to cutting-edge frontier AI models. The results demonstrate PriceBench's exceptional performance across all categories.

🤖 AI Language Models

Comparison with state-of-the-art AI models shows PriceBench's competitive edge:

🔬 Traditional Machine Learning Methods

Comparison with classical ML approaches highlights the advantage of specialized LLM training:

📈 Baseline Comparisons

👋 About Me

Hi there! I'm Sushant Nagi, a passionate developer who loves creating intelligent, scalable applications that push the boundaries of what's possible with modern technology.

📧 Contact Me

I'm always excited to connect with fellow developers, potential collaborators, or anyone interested in AI and technology!