Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit

CS245: Big Data Analytics (Fall 2025)

Final Project

Due Date: December 5, 2025

Instructions

Submission Guidelines

Submit your solutions as a PDF file through GradeScope via BruinLearn. Both typed and scanned handwritten submissions are acceptable, but illegible work will receive no credit. We strongly recom-mend using LATEX for your write-up.

Each student will have 24 total late hours that can be used flexibly across all assignments without penalty. No submissions will be accepted once these hours are exhausted.

Project Policy

Team Formation

Projects may be completed individually or in teams of up to four (4) students. Each team member is expected to contribute meaningfully to all stages of the project, including design, implementation, and documentation.

Project Options

Students may select one of the following three project options.

Option 1: LLM Agent Development (Recommended: AgentSociety Challenge Track)

Goal.   Develop an intelligent agent capable of performing multi-step reasoning, retrieval, or tool use in a simulated environment. Projects should showcase how large language models can autonomously reason, interact, and improve through contextual feedback.

Description.   Students are encouraged to participate in the AgentSociety Challenge—a research framework for building, simulating, and evaluating multi-agent systems that emulate realistic societal behaviors. The environment (websocietysimulator/) provides ready-to-use datasets from platforms such as Yelp, Amazon, and Goodreads. Each student or team will extend the provided base agents to design novel decision-making or reasoning strategies.

Core Tasks (Choose one of following).

• Track A – User Simulation: Implement an agent that models realistic user behaviors such as rating, reviewing, and preference formation.

• Track B – Recommendation: Develop an intelligent recommendation agent that adapts to diverse user preferences and optimizes engagement.

• Optional: Integrate both tracks to form an interactive user–recommender ecosystem.

Requirements.

• Implement at least one custom agent by subclassing the provided base classes (e.g., SimulationAgent or RecommendationAgent). You may choose one of above tasks.

• Incorporate advanced reasoning or decision-making strategies such as planning, memory, context retrieval, reflection, or self-improvement.

• Evaluate performance using the challenge’s metrics (e.g., HR@K, RMSE, sentiment alignment) and compare against baseline agents.

• Conduct ablation or comparative studies (e.g., with/without memory, varying prompts, or ex-ploration strategies).

• Submit (i) runnable code, (ii) experiment logs, and (iii) a written report detailing design rationale, results, and key insights, (iv) presentation slides.

Option 2: Open Project (Free Topic)

Goal. Propose and execute an independent research or engineering project related to machine learn-ing, natural language processing, or large language models. Projects should demonstrate technical originality, depth of understanding, and a clear experimental or analytical contribution.

Description.   Students may define their own project topics, ranging from theoretical explorations to applied system development. Possible directions include:

• Designing new training objectives, architectures, or fine-tuning methods for LLMs.

• Implementing or analyzing alignment or reinforcement learning from feedback methods (e.g., RLHF, DPO, GRPO, RLOO).

• Building domain-specific LLMs or retrieval-augmented generation (RAG) systems.

• Benchmarking reasoning or scientific problem-solving capabilities of open-source models.

• Exploring interpretability, safety, or fairness aspects of LLMs.

Proposal Requirement.   Each team must submit a 1–2 page proposal including:

• Problem statement and motivation.

• Background or related work.

• Proposed approach, datasets, and evaluation metrics.

• Expected outcomes and milestones.

Proposals will be reviewed for feasibility, technical depth, and relevance to the course objectives.

Grading Rubric

• 1. Technical Implementation (35%) Evaluate the correctness, completeness, and efficiency of the implemented system or experimental setup. For AgentSociety projects, this includes the ability of the agent to run successfully within the simulation environment and produce valid out-puts. For Open Projects, this includes correctness of algorithms, model training, and integration with datasets or APIs.

• 2. Innovation and Design Quality (15%) Assess the originality and creativity of the ap-proach. Projects should demonstrate thoughtful design choices, nontrivial strategies (e.g., reason-ing chains, adaptive context, learning signals), and clear motivation. Bonus points are awarded for integrating multiple components (e.g., reflection, planning, retrieval) or proposing new for-mulations.

• 3. Experimental Rigor and Evaluation (25%) Quality and depth of experiments, including baselines, ablations, and metric-based evaluation. Results should be reproducible, statistically meaningful, and well-interpreted. AgentSociety projects should report benchmark metrics (e.g., HR@K, RMSE, sentiment alignment); Open Projects should demonstrate sound evaluation pro-tocols (e.g., test sets, accuracy, reasoning metrics, or human evaluation).

• 4. Report and Analysis (20%) Clarity, organization, and analytical depth of the written report. The report should clearly describe problem formulation, system design, methodology, results, and key insights. Comparative discussions (what worked, what failed, why) are expected.

• 5. Reproducibility and Presentation (5%) Code quality, documentation, and ease of re-production. Proper README files, environment specifications, and example commands should be included. If a presentation or demo is required, clarity of explanation and ability to answer technical questions will also be considered.

Deadlines and Deliverables

• Project Proposal: Due by November 1, 2025. Each proposal should include project title, team members, selected option, and a one-paragraph summary. If you choose project 1, you do not need to submit project proposal.

• Project Presentation: December 1, 2025 and December 3, 2025. Both presentation sessions will be held online via Zoom. All team members are required to present their project.

• Final Submission: Due by December 5, 2025. Submission includes report, code, experiment logs, and presentation slides.

Computing Resources

Each student will receive a Google Cloud Platform (GCP) credit worth $50 to access GPU resources. Please note:

• GPU availability on GCP is not guaranteed during high-demand periods.

• Students are responsible for managing GPU access on their own, including using personal or departmental resources if needed.

• The signup link is GCP Credit. Please follow the instructions uploaded in Bruinlearn.

• If you need additional resources, note that Google Cloud Platform (GCP) offers $300 in free credits for every new account, which you may use as a backup plan.

General Notes

• Projects must reflect each team’s own original work and understanding.

• Use of pre-trained LLMs is encouraged where appropriate, but proper citation and documentation are required.

• Collaboration across teams is allowed for discussion, but code and report submissions must be unique.