DTS311TC Final Year Project Application of Smart Car for Line Following and Large Model Integration

Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit

DTS311TC Final Year Project

State Prediction:

Predicts the robot's next state based on its current state and control input.

(1)

x k|k-1：Predicted state at time t.

f(.)：Motion model, which maps the previous state and control input to the predicted state.

ut ：Control input, such as velocity or steering angle.

wt ：process noise, accounting for uncertainty in motion.

Kalman Gain:

Calculates the optimal weighting for the measurement update, balancing measurement noise.

(2)

PK|K—1  : Predicted covariance, representing the uncertainty in the state prediction.

Ht ：Jacobian matrix of the measurement model.

Rt ：Measurement noise covariance, accounting for sensor inaccuracies.

State Update:

Corrects the predicted state using the measurement to yield an updated estimate.

(3)

Zk：Actual measurement from the environment.

h（.）：Measurement model, mapping the predicted state to the measurement space.

These steps allow the robot to build a continuously updated map of the environment while estimating its position accurately. This is especially critical in retail settings where the layout may frequently change due to dynamic obstacles like customers and inventory adjustments.

2.2 Human-Robot Interaction (HRI) Using Large AI Models

Human-robot  interaction  in  retail  environments  plays  a  crucial  role  in  enhancing  customer experience. The use of large AI models, like "iFLYTEK Spark" or GPT-based architectures, has proven effective in understanding natural language and engaging in meaningful dialogues. Vaswani et  al.  (2017)  introduced  the  Transformer  architecture,  which  has  become  a  core  component  in modern language models, enabling them to understand and generate human-like responses. These models significantly enhance the communication capabilities of autonomous robots, allowing them to comprehend  customer needs, provide recommendations,  and record  item requests  seamlessly, thereby contributing to an interactive and personalized shopping experience.

The Transformer architecture relies on self-attention mechanisms that allow the model to weigh different parts of an input sequence, making it highly efficient at handling conversational context.

The self-attention mechanism is computed as follows:

(4)

Q, K , V：Query, key, and value matrices derived from the input sequence.

:   Scaled dot product, capturing relationships between input elements.

softmax（.）：Normalizes attention scores to focus on key tokens.

The Transformer also includes a feed-forward network at each position, which operates as follows:

(5)

x：Normalizes attention scores to focus on key tokens.

W1 , W2 ：Trainable weight matrices.

b1，b 2：Bias terms.

ReLU：Non - linear activation function, introducing complexity.

2.3 Line Following and Path Planning

Line following is a foundational aspect of robotic navigation, particularly in controlled environments such as warehouses and supermarkets. Traditional methods include using infrared sensors or cameras to  detect  lines,  while  recent   advancements   involve  deep  learning-based  methods  to   improve robustness. The RKNN (Rockchip Neural Network) platform provides an efficient solution for edge AI, facilitating real-time line following with high accuracy. Integrating RKNN with computer vision methods, as highlighted by Goodfellow et al. (2016) in their work on deep learning, has enabled smarter  and  more  reliable  navigation   systems  that  can  adapt  to  varying  lighting  and   surface conditions, making them suitable for real-world retail environments.

The line-following task can be modeled as an image classification or segmentation problem, where a convolutional  neural  network  (CNN)  processes  camera  images  to  detect  the  path.  The  RKNN platform  allows  the  deployment  of such  models  efficiently  on  edge  devices,  enabling  real-time decision-making directly on the vehicle.

Pooling Layer (e.g., Max Pooling):

(6)

x[i : i + k, j + j + k]：Size of the pooling window

k：Size of the pooling window

y[i, j]：Output value, representing the maximum value in the pooling region.

Pooling layers improve computational efficiency and focus the model's attention on the most critical features of the path, enabling robust navigation in dynamic retail environments.

3.Project Plans

The successful implementation of the smart car system for autonomous navigation and human-robot interaction  in  retail   environments  requires   a  well-structured  project  plan.  With  the  hardware components  already  available,  the  focus  will  shift  to  the  development,  testing,  and  deployment phases.   This   includes   integrating   navigation,    line-following,    and    large   AI    model-based communication  functionalities  into  a  cohesive  system.  Drawing  from  proven  methodologies  in autonomous  systems  (Zhang  &  Singh,  2017),  foundational  principles  of  deep  learning  (LeCun, Bengio, & Hinton, 2015), and advanced segmentation techniques (Shi & Malik, 2000), the project emphasizes robustness and efficiency to ensure reliable performance in real-world retail settings.

3.1 Smart Car System Design and Operational Framework

The proposed solution focuses on integrating autonomous navigation, human-robot interaction, and line-following capabilities to develop a smart car system that can navigate efficiently in a retail environment,  assist  customers,  and  retrieve  items  autonomously.  The  methodology  involves  the following steps:

3.1.1 SLAM-Based Navigation

To  enable  real-time  mapping  and  localization  in  a  dynamic  supermarket  environment,  SLAM (Simultaneous Localization and Mapping) will be implemented. The smart car will use LiDAR, IMU, and RGB cameras to build a real-time map of the environment and accurately determine its position. Path planning algorithms will then be developed to navigate through aisles and shelves, allowing the robot to move efficiently while avoiding obstacles.

The SLAM workflow, as depicted in Figure 1: SLAM Workflow Diagram, outlines the sequential processes involved in real-time mapping and localization. The system begins with data collection from sensors such as LiDAR and IMU, which is then processed through data association to match observations with existing map features. Tracks for new features are initialized, while outdated or irrelevant tracks and features are removed to maintain accuracy. The system predicts the smart car's next state based on control inputs like velocity and direction and dynamically updates the map to reflect changes in the environment. This iterative process ensures the smart car can adapt to dynamic environments  such  as  crowded  supermarkets  (Thrun,  Burgard,  &  Fox,  2005;  Engel,  Schöps,  & Cremers, 2014).

Figure 1:SLAM Workflow Diagram

The SLAM system will continuously update the map as the robot moves, enabling the car to adapt to dynamic  changes  in  the  environment.  This  approach  aligns  with  the  techniques  introduced  in LSD-SLAM, a method for monocular visual SLAM that is highly adaptable to large-scale settings (Engel,  Schöps,  & Cremers, 2014). Furthermore, leveraging ORB-SLAM offers an accurate and versatile alternative for monocular SLAM applications, which is particularly valuable for robust, real-time localization (Mur-Artal, Montiel, & Tardos, 2015). The integration of object detection algorithms such as YOLOv3 also enhances the smart car's ability to detect and avoid obstacles with high precision and low latency (Redmon & Farhadi, 2018), while multi-view 3D object detection supports effective environmental perception across multiple perspectives (Chen et al., 2017). This comprehensive integration ensures that the smart car can navigate through a constantly changing supermarket environment in real time (Thrun, Burgard, & Fox, 2005).

Figure 2 shows the system architecture, including the interaction between the SLAM, AI interaction, and RKNN modules, as well as the sensors used for mapping and localization.

Figure 2: System Architecture Diagram

3.1.2 Human-Robot Interaction Using AI Models

The  smart  car  will  be  integrated  with  the  large  AI  model  "iFLYTEK  Spark"  to  enable  natural language processing (NLP) for meaningful interaction with customers. This capability allows the car to understand spoken customer queries, such as requests for specific products or assistance with store navigation.  The   system  also  incorporates   speech  recognition   and  intent  detection  modules  to accurately interpret customer needs, leveraging the self-attention mechanisms in Transformer models to handle complex conversational context (Vaswani et al., 2017). Additionally, pre-trained language models, such as BERT, have proven effective in enhancing the precision of intent detection, further refining the interaction quality in customer queries (Devlin et al., 2019).

Once the intent is detected, the system generates contextual responses, enabling the robot to engage with customers in a personalized manner. This interaction model is supported by techniques like the GPT-3 model, which enhances contextual response generation through advanced language modeling capabilities (Brown et al., 2020). This interaction will be key to enhancing the shopping experience by enabling the smart car to provide real-time assistance in a natural, human-like way.

Figure 3 illustrates the flow of interaction from customer speech input to the car’s response, showing the process of speech recognition, intent detection, and contextual response generation.

Figure 3: AI Interaction Flowchart

3.1.3 Line Following Using RKNN

The  RKNN  (Rockchip  Neural  Network)  platform  will  be  used  to  implement  a  line-following mechanism for the smart car. A Convolutional Neural Network (CNN) will be trained to detect floor lines and guide the robot along predefined paths within the supermarket. This setup allows the robot to  follow  a  specific  route  while  avoiding  obstacles  and  other  disruptions,  with  CNNs  proving effective  for  feature  extraction  in  image-based  tasks  (Goodfellow,  Bengio,  &  Courville,  2016). Additionally, the integration of MobileNet, an efficient neural network for mobile and embedded vision applications, supports edge AI processing on resource-limited devices, allowing real-time line detection and path correction (Howard et al., 2017).

As   illustrated   in   Figure   4,   the   architecture   demonstrates   an   RKNN-based   framework   for time-dependent state prediction. RKNN (Rockchip Neural Network) provides an efficient platform for deploying deep learning models on edge devices, enabling real-time processing. In this setup, the system uses the current state   yr ，i（tj） and  control parameter μ（tj） as inputs, processes them through multiple neural network layers, and outputs the predicted state   yr ，i（tj+1 )    for the next time step.

This  architecture  leverages  RKNN's  efficient  computation  capabilities,  which   are  particularly suitable for dynamic system modeling and real-time control applications. RKNN provides a robust platform for deploying deep learning models on edge devices, enabling high-performance inference even  on  resource-constrained  hardware  (Rockchip,  2019).  This  design  ensures  accurate  and consistent state predictions, making it ideal for real-world dynamic environments.

Figure 4: MLP for Time-State Prediction

Using edge AI enables the decision-making process to be executed instantly, ensuring the car reacts swiftly  to  environmental  changes.  This  capability  is  essential  for  navigating  narrow  aisles  and following  complex  paths,  which  are  often  challenging  for  traditional  line-following  algorithms. Moreover, implementing transfer learning techniques on edge devices has shown promise in further improving model performance and adaptability to dynamic environments (Pan & Yang, 2010).

Figure 3.3 shows a diagram illustrating how the CNN model processes the camera feed to detect the path and guide the smart car along its designated route.

Figure 5: CNN Line Following Diagram

3.2 Experimental Design

To evaluate the performance of the proposed smart car system, a comprehensive experimental setup will be implemented, focusing on both individual modules—SLAM-based navigation, human-robot interaction,  and  line  following—and  the  integrated  system's  overall  functionality.  As  shown  in Figure 6: Task Map, a 4.5 × 4.5 m test area will be constructed. The experiment begins at point A, where the smart car initiates a dialogue with the user to obtain task details using the human-robot interaction module. After understanding and recording the requested item, the smart car employs the SLAM module to navigate autonomously to the yellow area, representing the item’s location. During this stage, the SLAM system ensures accurate localization and obstacle avoidance. Once the car reaches the yellow area and simulates item collection, it returns to A, switching to line-following mode. Finally, the car follows a predefined line from A to C to complete the task. This experimental design ensures a thorough evaluation of all system components under realistic conditions, providing valuable insights into their performance and integration.

Figure 6: Task Map

The performance of the smart car system will be evaluated through a comprehensive series of tests focusing  on  individual  modules—SLAM-based  navigation,  human-robot  interaction,  and  line following—as well as the overall integrated system.

l Phase 4: Evaluation & Improvement (Weeks 18-25)

Model evaluation and iterative improvements will start in Week 18. This phase includes testing the AI interaction model in a simulated supermarket environment, fine-tuning the line-following and navigation   algorithms,   and   comparing   different    configurations   to   optimize   overall    system performance.

l Phase 5: Summary (Weeks 26-28)

The final phase focuses on documentation, project presentation, and deployment preparation. A draft of the project report will be created, followed by the final presentation. The project will conclude with the submission of the dissertation and deployment plan for real-world testing.

3.3.2 Gantt Chart

The Gantt chart below provides a timeline for each phase and task, distributed across two semesters. This breakdown helps to track progress and manage project timelines effectively.

Figure3.3.2:Project Timeline for Smart Car Development