Fixing `/api/log` Parsing Issues In `byos_laravel` For Accurate Log Saving

Jul 28, 2025 by ADMIN 75 views

`/api/log` Parsed Incorrectly Resulting in No Logs Being Saved

Hey guys! 👋 Let's dive into an issue where the /api/log endpoint wasn't parsing correctly, leading to logs not being saved. This problem was discovered while using a BYOD (Bring Your Own Device) setup with a mostly stock firmware and byos_laravel to proxy the cloud service, along with plugins for other devices. The user noticed that the logs page in their byos_laravel instance was consistently empty, which prompted further investigation.

Identifying the Issue: Log Submissions and Empty Tables

To identify the root cause, the user examined the ESP32 console logs and found that logs were being submitted. Here's an example of what the log entries looked like:

I: src/bl.cpp [1982]: log string - {"created_at":0,"id":1,"message":"Failed to resolve hostname on attempt 1","source_line":598,"source_path":"src/bl.cpp","wifi_signal":-74,"wifi_status":"connected","refresh_rate":0,"sleep_duration":0,"firmware_version":"1.6.0-wip","special_function":"none","battery_voltage":4.2,"wake_reason":"timer","free_heap_size":252116,"max_alloc_size":110592,"retry":1},{"created_at":0,"id":2,"message":"Failed to resolve hostname on attempt 2","source_line":598,"source_path":"src/bl.cpp","wifi_signal":-76,"wifi_status":"connected","refresh_rate":0,"sleep_duration":0,"firmware_version":"1.6.0-wip","special_function":"none","battery_voltage":4.2,"wake_reason":"timer","free_heap_size":252032,"max_alloc_size":110592,"retry":1},{"created_at":0,"id":3,"message":"Failed to resolve hostname on attempt 3","source_line":598,"source_path":"src/bl.cpp","wifi_signal":-78,"wifi_status":"connected","refresh_rate":0,"sleep_duration":0,"firmware_version":"1.6.0-wip","special_function":"none","battery_voltage":4.2,"wake_reason":"timer","free_heap_size":252116,"max_alloc_size":110592,"retry":1},{"created_at":0,"id":4,"message":"Failed to resolve hostname on attempt 4","source_line":598,"source_path":"src/bl.cpp","wifi_signal":-73,"wifi_status":"connected","refresh_rate":0,"sleep_duration":0,"firmware_version":"1.6.0-wip","special_function":"none","battery_voltage":4.2,"wake_reason":"timer","free_heap_size":252116,"max_alloc_size":110592,"retry":1}
I: src/bl.cpp [1983]: need to send the log
I: src/api-client/submit_log.cpp [11]: [HTTPS] begin /api/log ...
I: lib/trmnl/include/http_client.h [25]: ==== withHttp() https://[hidden]/api/log
I: src/api-client/submit_log.cpp [25]: [HTTPS] POST...
I: src/api-client/submit_log.cpp [37]: Send log - |{"logs":[{"created_at":0,"id":1,"message":"Failed to resolve hostname on attempt 1","source_line":598,"source_path":"src/bl.cpp","wifi_signal":-74,"wifi_status":"connected","refresh_rate":0,"sleep_duration":0,"firmware_version":"1.6.0-wip","special_function":"none","battery_voltage":4.2,"wake_reason":"timer","free_heap_size":252116,"max_alloc_size":110592,"retry":1},{"created_at":0,"id":2,"message":"Failed to resolve hostname on attempt 2","source_line":598,"source_path":"src/bl.cpp","wifi_s...
I: src/api-client/submit_log.cpp [55]: [HTTPS] POST OK, code: 200
I: lib/trmnl/src/stored_logs.cpp [115]: Cleared 4 stored logs

The logs indicated a 200 OK response, meaning the server received the data. However, the Laravel logs and the SQLite database table (device_logs) were empty. This discrepancy suggested an issue within the byos_laravel application, specifically in how it was processing the incoming log data. The database being empty after successful log submissions pointed to a parsing or data handling problem on the server side. This required a deeper look into the byos_laravel codebase to identify where the incoming JSON payload was being mishandled, preventing the logs from being correctly stored in the database. This investigation led to examining the routes and controllers responsible for handling the /api/log endpoint.

Diagnosing the Problem: Incorrect JSON Parsing

The user pinpointed the potential issue to a specific line in the api.php routes file of byos_laravel (this line). The original code was using $request->json('log.logs_array', []); to extract the logs from the JSON payload. The problem here is that this code assumes the JSON structure has a log key with a nested logs_array key, but this structure may not be consistent across different firmware versions or updates. The user proposed a simpler solution of changing this line to $request->json('logs', []);. This adjustment suggests that the expected JSON structure should directly contain a logs array at the top level, without the nested log object. This approach would align better with the actual JSON structure observed in the logs and potentially resolve the parsing issue.

To further diagnose the problem, the user traced back the changes in the firmware's logging format. By examining the commit history, they found the commit where the log format was modified (here). This commit was included in the v1.5.10 release, indicating that the change in log format occurred with this version. This discovery is crucial because it implies that the correct way to parse the logs might depend on the device's firmware version. Devices running older firmware might use a different JSON structure than those running v1.5.10 or later. Therefore, a robust solution would need to dynamically adjust the parsing logic based on the firmware version of the device sending the logs. This version-aware parsing would ensure compatibility across different firmware releases and prevent the issue of logs not being saved due to incorrect parsing.

Proposing a Solution: Version-Aware JSON Parsing

Based on the investigation, the user suggested that a correct solution should dynamically choose the JSON key based on the device's firmware version. This approach acknowledges that the log format might have changed between firmware versions, specifically around the v1.5.10 release. To implement this, the byos_laravel application would need to determine the firmware version of the device sending the log data and then use the appropriate JSON key to extract the logs. For example, if the firmware version is older than v1.5.10, the application might use $request->json('log.logs_array', []);. However, if the firmware version is v1.5.10 or newer, it would use $request->json('logs', []);. This conditional parsing ensures that the application can handle logs from devices running different firmware versions correctly.

To implement this solution, the application would need to access the firmware version information from the incoming request. This could be achieved by including the firmware version in the request headers or as part of the JSON payload itself. Once the firmware version is retrieved, a simple conditional statement can be used to determine which JSON key to use for parsing the logs. This approach not only resolves the immediate issue of logs not being saved but also makes the application more resilient to future changes in the log format. By adapting to the firmware version, the application avoids hardcoding assumptions about the JSON structure and can gracefully handle logs from a wider range of devices and firmware releases. This proactive approach ensures the reliability and maintainability of the logging functionality in the byos_laravel application.

Implementing the Fix: Practical Steps

To implement the version-aware JSON parsing, you can follow these steps:

Extract Firmware Version: Modify the API endpoint to extract the firmware version from the request. This could be from a header, a query parameter, or within the JSON payload itself. For example, if the firmware version is sent in the header X-Firmware-Version, you can access it using $request->header('X-Firmware-Version'). Ensure that the firmware version is reliably included in the requests from the devices.
Conditional Parsing: Use a conditional statement to parse the JSON based on the firmware version. Here’s an example of how this can be implemented in the api.php route:
```
$firmwareVersion = $request->header('X-Firmware-Version');
$logs = [];

if (version_compare($firmwareVersion, '1.5.10', '>=')) {
    $logs = $request->json('logs', []);
} else {
    $logs = $request->json('log.logs_array', []);
}

// Process the $logs array
foreach ($logs as $log) {
    // ... logic to save the log to the database ...
}
```
In this example, version_compare is used to compare the extracted firmware version with 1.5.10. If the firmware version is greater than or equal to 1.5.10, it uses $request->json('logs', []) to parse the JSON. Otherwise, it uses $request->json('log.logs_array', []). This ensures that the correct parsing logic is applied based on the firmware version.
Process Logs: After parsing the logs, iterate through the $logs array and save each log entry to the database. This involves creating a new DeviceLog model instance for each log entry and populating it with the relevant data from the log. Ensure that the database schema matches the structure of the log entries, including fields for created_at, id, message, source_line, source_path, wifi_signal, wifi_status, and other relevant information.
Testing: Thoroughly test the implementation with devices running different firmware versions. This is crucial to ensure that the conditional parsing logic works correctly and that logs are being saved for all devices. Use a variety of devices and firmware versions to cover all possible scenarios. Verify that the logs are being saved to the database correctly and that the log entries contain the expected data.

By following these steps, you can implement a version-aware JSON parsing solution that addresses the issue of logs not being saved in byos_laravel. This ensures compatibility across different firmware versions and provides a robust and maintainable logging solution.

Conclusion: Ensuring Robust Log Handling

In conclusion, the issue of /api/log parsing incorrectly highlights the importance of handling data format changes gracefully, especially when dealing with different versions of firmware or software. By implementing a version-aware JSON parsing strategy, we can ensure that the byos_laravel application correctly processes logs from devices running various firmware versions. This approach not only resolves the immediate problem of missing logs but also enhances the long-term reliability and maintainability of the application.

The key takeaway is to avoid hardcoding assumptions about data formats and instead, adopt a flexible approach that can adapt to changes. This can be achieved by incorporating version information into the data processing logic and using conditional statements to handle different formats. By doing so, we can build more robust and resilient systems that can withstand the test of time and changing requirements. Guys, always remember to prioritize flexibility and adaptability in your code to minimize future issues and ensure a smooth user experience across all devices and firmware versions.