diff --git a/docs/01-basic/report.md b/docs/01-basic/report.md new file mode 100644 index 0000000..21b9dfe --- /dev/null +++ b/docs/01-basic/report.md @@ -0,0 +1,45 @@ +# 问答题报告 + +### (Q1.1) + +在给出的代码框架 `Parser` 中： + +* 哪条语句或哪几条语句将日志按逗号进行分割？代码中，我们是如何指定每一行的第几个字段代表何种意义的？ +> using var csv = new CsvReader(logFile, config);配合循环使用的csv.GetRecords()； +>在LogRecordMap中一一指定对应 + +* 在对日志中 JSON 格式的 `message` 字段进行读取时，我们是在哪个方法内用哪几条语句判断这一行日志的种类（Call / Request / Internal）的？ +> LineParser 类的 ParseLine 方法，if(root.TryGetProperty("event", out var eventElement))语句，再用swith判断 + +* 在确定了日志种类后，我们是调用了哪个库方法对 JSON 进行解析的？ +> System.Text.Json 库中的 JsonSerializer.Deserialize(...) 方法 + +* 进一步，我们的框架代码是如何防止日志中有字段缺失的？（例如所给的 Call 日志的 `message` 中缺失 `request_id` 字段） +> 在每个属性前面都强制加上了 [property: JsonRequired] 特性标签 + +* 更进一步，日志中的 JSON 的键是 `abc-def` 命名法（称为烤串命名法），而我们的解析结果却是放在 `AbcDef` 命名法（称为大驼峰命名法）的属性里，我们的框架代码中是如何告诉 JSON 解析器完成这一命名法转换的？ +> 框架代码事先创建了一个名为 options 的配置变量，并在其中设置了 JsonNamingPolicy.KebabCaseLower 这一规则。在调用 JsonSerializer.Deserialize 提取数据时，代码将这个 options 作为参数交给了解析工具。 + +--- + +### (Q1.2) + +以一个 Call 事件的解析结果为例，当调用 `KeyValueVisitor` 的 `Dump` 方法后，都有哪些方法被调用？请补充完整如下的方法调用链（.NET 内置库无需写出）： + ++ `Dictionary KeyValueVisitor.Dump(LogEntry entry)` ++ > TResult CallLogEntry.Accept(ILogEntryVisitor visitor) ++ > Dictionary KeyValueVisitor.Visit(CallLogEntry entry) + + +--- + +### (Q1.3) + +本次作业中，你是否使用了 AI？根据你的使用情况，在以下 (Q1.3.a) (Q1.3.b) 两个问题中选择一题作答： + + +#### (Q1.3.b) +如果使用了 AI，你给予 AI 的提示词是什么？你认为 AI 给出的解答、你完全凭借传统搜索引擎以及自己的能力能够写出的解答之间，AI 的解答比你好在哪？AI 又有哪些解答是存在问题的，或者至少是不如你自己的解答的？给出你的理由。 ++ > 提示词上，在完成代码部分，我只是把ai当做搜索引擎使用，去解释一些我看不懂的C#内置函数，然后再用自己能力补充代码，不过在q1.1中第1，4，5个问题，我确实完全不了解c#内置的库是什么，让ai先给出了解答再自己借助ai去了解； ++ > 代码中，ai的解答和我几乎相同；在问答题中，ai的解答比我更具有专业性，对整个项目的把握比我更透彻 ++ > 我让只让ai给出了q1.1中1，4，5题超出我能力范围之外的解答，这些解答核实没发现什么大问题，其他的ai解答和我大致相同，我暂时未发现ai明显不如自己解答的情况。 \ No newline at end of file diff --git a/docs/02-multithreading/image-1.png b/docs/02-multithreading/image-1.png new file mode 100644 index 0000000..5382b05 Binary files /dev/null and b/docs/02-multithreading/image-1.png differ diff --git a/docs/02-multithreading/image.png b/docs/02-multithreading/image.png new file mode 100644 index 0000000..091cbb7 Binary files /dev/null and b/docs/02-multithreading/image.png differ diff --git a/docs/02-multithreading/report.md b/docs/02-multithreading/report.md new file mode 100644 index 0000000..cfe4a01 --- /dev/null +++ b/docs/02-multithreading/report.md @@ -0,0 +1,57 @@ +## 任务 2.3：实现前端交互逻辑 (Console UI) + +### 1. 功能介绍 +本任务在 `Program.cs` 中实现了控制台交互界面，主要完成了以下三个功能： + +* **指定文件分析 (AnalyzeFiles)**： + 接收用户输入的以逗号分隔的文件名，自动去除多余空格和空项。通过安全的方式读取用户输入的线程数，如果遇到输入格式错误或者后台任务冲突，会通过循环把用户留在当前步骤要求重新输入，防止程序崩溃。 +* **全目录分析 (AnalyzeAll)**： + 只需读取用户输入的线程数，在校验输入合法后，直接交给后台并发分析整个文件夹里的所有日志。 +* **查询结果 (GetAnalysisResult)**： + 获取用户输入的文件名，调用后台查询。如果文件还没有分析，提示未分析；如果分析失败，打印具体的报错信息；如果分析成功，则调用 `KeyValueVisitor.Dump` 方法，把底层的日志数据转换成键值对字典，并逐行清晰地打印到屏幕上。 + +### 2. 运行效果截图 + +![alt text](image.png) + + +![alt text](image-1.png) + +### 3. 问答题 + +#### (Q2.1) 临界区与数据竞争理解 + +* **`WorkQueue` 类中的共享变量有哪些？是通过什么保护其免于数据竞争（data race）呢？** + > 答：_items和_isCompleted；运用lock（）形成互斥锁 + > + > + +* **`LogFileAnalyzer` 类中的共享变量有哪些？是通过什么保护其免于数据竞争呢？** + > 答：_currentDirectory，_isAnalyzing，_logFiles，_analysisResults；定义 _syncRoot作为互斥锁对象来保护 + > + > + > + +* **如果条件变量的判断条件使用了 `if` 判断而非 `while` 判断，当出现了虚假唤醒现象时（在类 UNIX 系统中，由于 UNIX 信号等机制，即使没有人调用过 `signal` 或 `broadcast`，处于 `wait` 当中的条件变量也可能被唤醒），会出现什么后果？结合无限仓库容量的生产者消费者问题简单叙述一下。** + > 答：这样的话会导致消费者在被异常唤醒时跳过if检查直接强行尝试从空仓库中取商品，，进而导致程序发生崩溃。 + > + > + > + +#### (Q2.2) 目录扫描逻辑 + +* **在给出的代码框架 `LogFileAnalyzer` 中，那一段代码扫描了给定的目录中的全部 `.log` 后缀的日志文件？假使给定的需求是不但要扫描给定目录中的日志文件，还要递归地获取给定的目录的全部子目录、子子目录……内的日志文件，应当如何做（简要回答即可）？**[cite: 6] + > 答：ChangeDirectory 方法中Directory.EnumerateFiles(directoryPath, "*.log", SearchOption.TopDirectoryOnly)； + > SearchOption.TopDirectoryOnly 修改为 SearchOption.AllDirectories + > + > + +#### (Q2.3) AI 使用情况调查 + +*本次作业中，你是否使用了 AI？根据你的使用情况，在以下 (Q2.3.a) (Q2.3.b) 两个问题中选择一题作答即可：* + +* **(Q2.3.b) 如果使用了 AI，你给予 AI 的提示词是什么？你对 AI 的使用是询问 AI 一些接口的用法或是在某处的写法，还是让 AI 帮你写一部分作业代码，又或是让 AI 给你讲解代码框架？AI 的解答是否出现过错误（如果有，是哪些）？你认为本节的难度是偏低、适中，还是偏高？** + > 答：我给予ai提示词更多是搜索性质与提示讲解性质，并没有让ai直接给出过代码。因为本节难度我认为对于之前从未接触过c#的人来说难度比较高，c#的一堆内置函数单凭自己很难看懂，更何况有时候会一下子牵扯到多个代码文件，没有ai的辅助梳理很容易忘记要干什么。 + > + > + > \ No newline at end of file diff --git a/docs/03-async-grpc/image-1.png b/docs/03-async-grpc/image-1.png new file mode 100644 index 0000000..bdb7297 Binary files /dev/null and b/docs/03-async-grpc/image-1.png differ diff --git a/docs/03-async-grpc/image.png b/docs/03-async-grpc/image.png new file mode 100644 index 0000000..f4d7d7a Binary files /dev/null and b/docs/03-async-grpc/image.png differ diff --git a/docs/03-async-grpc/report.md b/docs/03-async-grpc/report.md new file mode 100644 index 0000000..86e6e7c --- /dev/null +++ b/docs/03-async-grpc/report.md @@ -0,0 +1,38 @@ +# 异步 gRPC 日志分析系统实现报告 + +## 一、 功能介绍 + +本项目在之前本地日志分析器的基础上，成功升级为了基于 gRPC 框架的“客户端-服务端”网络架构。主要实现了以下核心功能： + +1. **服务端日志处理代理 (AgentSession)**： + - 实现了响应各种网络请求的方法。 + - 能够将底层系统抛出的异常（如路径非法、并发度为负数等）拦截，并转化为标准的网络状态码（如 `INVALID_ARGUMENT`, `DIRECTORY_NOT_FOUND` 等）返回给客户端。 + - 针对结果获取，实现了服务端的流式（Streaming）返回逻辑，先发送全局状态（Header），再循环发送具体的日志内容（LogEntry）。 + +2. **远程异步客户端 (RemoteCli)**： + - 全面采用了 `async/await` 异步编程模型，保证在等待网络响应时控制台不会卡死。 + - **切换目录 (Change Directory)**：向服务器发送路径，并接收服务器返回的当前绝对路径和文件列表。 + - **显示文件 (Show Log Files)**：通过网络请求获取当前远端工作目录下的所有日志文件。 + - **执行分析 (Analyze)**：支持全局分析和部分文件分析，将用户输入的并发度等参数打包发送，并根据服务器返回的状态码判断任务是否成功启动。 + - **流式接收结果 (Get Analysis Result)**：建立流式通道，根据服务器发回的数据包类型（Header 或是具体的日志行），分别进行状态打印和逐行解析展示。 + +## 二、图片展示 +![alt text](image.png) +![alt text](image-1.png) +（注：中间有个todo1.2是因为切换分支后没有及时同步第一讲的功能，不过不影响本讲内容实现） + +## 三、问题回答 +**问题 3.1：你认为，你在开发网络应用程序，与你在以往开发非网络应用程序的区别在哪里？网络应用程序的开发存在哪些额外的难点？存在哪些额外的复杂之处？** + +**答：** +> 核心区别在于数据通信： 本地程序直接在内存中传递对象，而网络程序必须将数据打包 +> 主要难点在于异步逻辑： 网络通信存在延迟，为了不让程序等待卡死，必须使用 async/await 等异步编程模型 +> 额外复杂之处在于极高的容错要求： 服务端作为常驻程序，绝不能因为用户的错误输入或内部小错误而崩溃。 + + + +**问题 3.2：本次作业中，你是否使用了 AI？根据你的使用情况，在以下 (Q3.2.a) (Q3.2.b) 两个问题中选择一题作答：** + +**(Q3.2.b) 如果使用了 AI，你给予 AI 的提示词是什么？你对 AI 的使用是询问 AI 一些接口的用法、gRPC 的使用，或是在某处的写法，还是让 AI 帮你写一部分作业代码，又或是让 AI 给你讲解代码框架？AI 的解答是否出现过错误（如果有，是哪些）？你从 AI 那里是否得知了一些关于异步，或是 gRPC 等原本你不知道或是难以理解的知识？** + +>我对ai的使用主要来自询问我看不懂的语句语法，让ai给我讲解作业中的代码框架。ai的解答出现过错误，比如一开始都没有使用异步编程模型等低级错误，但是人工一眼就看得出来。我使用ai主要就是补充知识，所以我的确获得了原本不知道的知识。 \ No newline at end of file diff --git a/src/LocalCli/LocalCli.csproj b/src/LocalCli/LocalCli.csproj index 0ad60d9..ea7e02f 100644 --- a/src/LocalCli/LocalCli.csproj +++ b/src/LocalCli/LocalCli.csproj @@ -18,4 +18,13 @@ + + + + + runtime; build; native; contentfiles; analyzers; buildtransitive + all + + + diff --git a/src/LocalCli/Program.cs b/src/LocalCli/Program.cs index 17b30db..03cec08 100644 --- a/src/LocalCli/Program.cs +++ b/src/LocalCli/Program.cs @@ -1,4 +1,5 @@ -using LogAnalyzer; +using System.Net; +using LogAnalyzer; using LogParser.Visitors; namespace LocalCli @@ -112,22 +113,149 @@ 6. Exit. private static void ShowLogFiles(LogFileAnalyzer analyzer) { - throw new NotImplementedException("T2.3"); + var files=analyzer.GetLogFiles(); + if (files.Count == 0) + { + Console.WriteLine("No log files found in the current directory."); + return; + } + int index=1; + foreach (var file in files) + { + Console.WriteLine($"{index}.{file}"); + index++; + } } private static void AnalyzeFiles(LogFileAnalyzer analyzer) - { - throw new NotImplementedException("T2.3"); + { while(true){ + Console.WriteLine("Please input log files to analyze (separated by comma):"); + var str=Console.ReadLine(); + if (str is null) + { + return; + } + var selectedFiles = str.Split(',', StringSplitOptions.RemoveEmptyEntries).Select(f => f.Trim()).ToArray(); + if (selectedFiles.Length == 0) + { + Console.WriteLine("No files specified, please try again."); + continue; + } + int degree = 0; + while (true) + { + Console.WriteLine("Please input degree of parallelism:"); + var degreeStr = Console.ReadLine(); + if (degreeStr is null) return; + if (!int.TryParse(degreeStr, out degree)) + { + Console.WriteLine("Invalid number, please try again."); + continue; + } + break; + } + try + { + analyzer.AnalyzeFiles(degree,selectedFiles); + break; + } + catch (InvalidOperationException) + { + Console.WriteLine("Analysis is already running, please wait."); + break; + } + catch(ArgumentException) + { + Console.WriteLine("Invalid files, please try again."); + continue; + } + } + } + private static void AnalyzeAll(LogFileAnalyzer analyzer) { - throw new NotImplementedException("T2.3"); + while (true) + { + Console.WriteLine("Please input degree of parallelism:"); + var degreeStr = Console.ReadLine(); + if (degreeStr is null) + { + return; + } + if (!int.TryParse(degreeStr, out int degree)) + { + Console.WriteLine("Invalid number, please try again."); + continue; + } + try + { + analyzer.AnalyzeAll(degree); + break; + } + catch (InvalidOperationException) + { + Console.WriteLine("Analysis is already running, please wait."); + break; + } + catch (ArgumentException) + { + Console.WriteLine("Invalid degree of parallelism, please try again."); + continue; + } + } } private static void GetAnalysisResult(LogFileAnalyzer analyzer) { - throw new NotImplementedException("T2.3"); + while (true) + { + Console.WriteLine("Please input log file name:"); + var fileName = Console.ReadLine(); + if (fileName is null) + { + return; + } + fileName = fileName.Trim(); + if (fileName.Length == 0) + { + Console.WriteLine("Invalid file name, please try again."); + continue; + } + if (!analyzer.TryGetAnalysisResult(fileName, out var result)) + { + Console.WriteLine("File not found, please try again."); + continue; + } + if (result is null) + { + Console.WriteLine("File not found, please try again."); + continue; + } + switch (result.State) + { + case AnalysisState.NotAnalyzed: + Console.WriteLine("Not analyzed."); + break; + case AnalysisState.Failed: + Console.WriteLine($"Failed: {result.ErrorMessage}"); + break; + case AnalysisState.Succeeded: + var visitor = new KeyValueVisitor(); + foreach (var entry in result.Entries) + { + var logDict = visitor.Dump(entry); + foreach (var kvp in logDict) + { + Console.WriteLine($"{kvp.Key}: {kvp.Value}"); + } + } + break; + + } + break; + } } } } diff --git a/src/LogAnalyzer/LogFileAnalyzer.cs b/src/LogAnalyzer/LogFileAnalyzer.cs index c3e7691..7daf113 100644 --- a/src/LogAnalyzer/LogFileAnalyzer.cs +++ b/src/LogAnalyzer/LogFileAnalyzer.cs @@ -142,6 +142,7 @@ public void AnalyzeFiles(int degreeOfParallelism, IEnumerable fileNames) * Set _isAnalyzing */ // TODO: T2.2 + _isAnalyzing = true; } try @@ -155,6 +156,10 @@ public void AnalyzeFiles(int degreeOfParallelism, IEnumerable fileNames) * Remember to lock _syncRoot to prevent data race */ // TODO: T2.2 + lock (_syncRoot) + { + _isAnalyzing = false; + } } } @@ -169,7 +174,14 @@ private void RunWorkers(int degreeOfParallelism, IReadOnlyList fileLis * Filter unparsed files. * If there is an unknown file, throw System.InvalidOperationException. */ - throw new NotImplementedException("TODO: T2.2"); + if (!_analysisResults.ContainsKey(file.Name)) + { + throw new InvalidOperationException($"Unknown file: {file.Name}"); + } + if (_analysisResults[file.Name].State == AnalysisState.NotAnalyzed) + { + logFilesToParse.Add(file); + } } } @@ -184,7 +196,11 @@ private void RunWorkers(int degreeOfParallelism, IReadOnlyList fileLis * Enqueue log files */ // TODO: T2.2 - + foreach (var file in logFilesToParse) + { + queue.Enqueue(file); + } + queue.CompleteAdding(); degreeOfParallelism = Math.Max(Math.Min(degreeOfParallelism, logFilesToParse.Count), 1); var workers = new Thread[degreeOfParallelism]; for (int i = 0; i < degreeOfParallelism; i++) @@ -195,12 +211,19 @@ private void RunWorkers(int degreeOfParallelism, IReadOnlyList fileLis * Create and start threads to run `WorkerMain` */ // TODO: T2.2 + workers[i] = new Thread(() => WorkerMain(workerId, queue)); + workers[i].Name = threadName; + workers[i].Start(); } /* * Wait for (join) all threads to end */ // TODO: T2.2 + foreach (var worker in workers) + { + worker.Join(); + } } private void WorkerMain(int workerId, WorkQueue queue) @@ -213,19 +236,38 @@ private void WorkerMain(int workerId, WorkQueue queue) try { // Parse file - throw new NotImplementedException("TODO: T2.2"); + using var reader = new StreamReader(file.FullName); + var entries = parser.Parse(reader); + result = new AnalysisResult( + FileName: file.Name, + FullName: file.FullName, + State: AnalysisState.Succeeded, + Entries: entries.ToList(), + ErrorMessage: null, + WorkerId: workerId + ); } catch (Exception ex) { // Save exception message to result - throw new NotImplementedException("TODO: T2.2"); + result = new AnalysisResult( + FileName: file.Name, + FullName: file.FullName, + State: AnalysisState.Failed, + Entries: Array.Empty(), + ErrorMessage: ex.Message, + WorkerId: workerId + ); } /* * Save parse result. * [!Important] Remember to lock _syncRoot to prevent data race. */ - throw new NotImplementedException("TODO: T2.2"); + lock (_syncRoot) + { + _analysisResults[file.Name] = result; + } } } } diff --git a/src/LogAnalyzer/WorkQueue.cs b/src/LogAnalyzer/WorkQueue.cs index 23055a5..d1f6112 100644 --- a/src/LogAnalyzer/WorkQueue.cs +++ b/src/LogAnalyzer/WorkQueue.cs @@ -20,17 +20,39 @@ public bool IsCompleted public void Enqueue(T item) { - throw new NotImplementedException("TODO: T2.1"); + lock (_items) + { + _items.Enqueue(item); + Monitor.Pulse(_items); + } + } public bool TryDequeue([NotNullWhen(true)] out T? item) { - throw new NotImplementedException("TODO: T2.1"); + lock (_items) + { + while (_items.Count == 0) + { + if (_isCompleted == true) + { + item=default; + return false; + } + Monitor.Wait(_items); + } + item=_items.Dequeue(); + return true; + } } public void CompleteAdding() { - throw new NotImplementedException("TODO: T2.1"); + lock (_items) + { + _isCompleted=true; + Monitor.PulseAll(_items); + } } } } diff --git a/src/LogAnalyzerAgent/Applications/AgentSession.cs b/src/LogAnalyzerAgent/Applications/AgentSession.cs index 2531f22..862a55c 100644 --- a/src/LogAnalyzerAgent/Applications/AgentSession.cs +++ b/src/LogAnalyzerAgent/Applications/AgentSession.cs @@ -79,22 +79,166 @@ public Task GetLogFiles(Empty empty, CancellationToken canc public Task ChangeDirectory(ChangeDirectoryRequest request, CancellationToken cancellationToken) { - throw new NotImplementedException("TODO: T3.1"); + var response=new ChangeDirectoryResponse(); + try + { + bool exists = _analyzer.ChangeDirectory(request.DirectoryPath); + if (!exists) + { + response.Status=new OperationStatusMessage + { + Success=false, + Code=AgentErrorCode.DirectoryNotFound, + Message="Directory not exists." + }; + } + else + { + response.Status=CreateNoErrorOperationStatus(); + response.CurrentDirectory=_analyzer.CurrentDirectory??""; + response.FileNames.AddRange(_analyzer.GetLogFiles()); + } + } + catch(ArgumentException ex) + { + response.Status = new OperationStatusMessage + { + Success = false, + Code = AgentErrorCode.InvalidArgument, + Message = "Directory illegal: " + ex.Message + }; + } + catch(Exception ex) + { + response.Status=CreateInternalErrorOperationStatus(ex); + _logger.LogError(ex, "An error occurred while changing directory."); + } + return Task.FromResult(response); } public Task AnalyzeAll(AnalyzeAllRequest request, CancellationToken cancellationToken) { - throw new NotImplementedException("TODO: T3.1"); + var response = new AnalyzeAllResponse(); + try + { + _analyzer.AnalyzeAll(request.DegreeOfParallelism); + response.Status = CreateNoErrorOperationStatus(); + } + catch(InvalidOperationException ex) + { + response.Status=new OperationStatusMessage + { + Success=false, + Code=AgentErrorCode.InvalidOperation, + Message="Analysis is already running: " + ex.Message + }; + } + catch(ArgumentException ex) + { + response.Status=new OperationStatusMessage + { + Success=false, + Code=AgentErrorCode.InvalidArgument, + Message = "Invalid degree of parallelism: " + ex.Message + }; + } + catch(Exception ex) + { + response.Status=CreateInternalErrorOperationStatus(ex); + _logger.LogError(ex, "An error occurred while analyzing all files."); + } + return Task.FromResult(response); } public Task AnalyzeFiles(AnalyzeFilesRequest request, CancellationToken cancellationToken) { - throw new NotImplementedException("TODO: T3.1"); + var response=new AnalyzeFilesResponse(); + try + { + _analyzer.AnalyzeFiles(request.DegreeOfParallelism,request.FileNames.ToArray()); + response.Status=CreateNoErrorOperationStatus(); + } + catch(InvalidOperationException ex) + { + response.Status=new OperationStatusMessage + { + Success=false, + Code=AgentErrorCode.InvalidOperation, + Message="Analysis is already running: " + ex.Message + }; + } + catch (ArgumentException ex) + { + response.Status = new OperationStatusMessage + { + Success = false, + Code = AgentErrorCode.InvalidArgument, + Message = "Invalid files or degree of parallelism: " + ex.Message + }; + } + catch (Exception ex) + { + response.Status = CreateInternalErrorOperationStatus(ex); + _logger.LogError(ex, "An error occurred while analyzing specified files."); + } + + return Task.FromResult(response); } public IReadOnlyList GetAnalysisResult(GetAnalysisResultRequest request, CancellationToken cancellationToken) { - throw new NotImplementedException("TODO: T3.1"); + var responseList = new List(); + try + { + if(!_analyzer.TryGetAnalysisResult(request.FileName,out var result) || result == null) + { + responseList.Add(new GetAnalysisResultResponse + { + Status =new OperationStatusMessage + { + Success=false, + Code=AgentErrorCode.FileNotFound, + Message="file not Found." + } + }); + return responseList; + } + var headerResponse =new GetAnalysisResultResponse + { + Status=CreateNoErrorOperationStatus(), + Header =new AnalysisResultHeaderMessage + { + FileName= request.FileName, + State=GrpcTypeConverter.ConvertToGrpc(result.State) + } + }; + if (!string.IsNullOrEmpty(result.ErrorMessage)) + { + headerResponse.Header.ErrorMessage=result.ErrorMessage; + } + responseList.Add(headerResponse); + if (result.State == AnalysisState.Succeeded) + { + foreach(var entry in result.Entries) + { + var LogEntryResponse=new GetAnalysisResultResponse + { + Status=CreateNoErrorOperationStatus(), + LogEntry =GrpcTypeConverter.ConvertToGrpc(entry), + }; + responseList.Add(LogEntryResponse); + } + } + + } + catch (Exception ex) + { + var errorResponse= new GetAnalysisResultResponse(); + errorResponse.Status=CreateInternalErrorOperationStatus(ex); + responseList.Add(errorResponse); + _logger.LogError(ex, "An error occurred while getting analysis result."); + } + return responseList; } } } diff --git a/src/LogAnalyzerAgent/Services/AgentService.cs b/src/LogAnalyzerAgent/Services/AgentService.cs index 591dcad..2be3f4e 100644 --- a/src/LogAnalyzerAgent/Services/AgentService.cs +++ b/src/LogAnalyzerAgent/Services/AgentService.cs @@ -29,27 +29,32 @@ public override Task GetAgentStatus(Empty empty, ServerC public override Task ChangeDirectory(ChangeDirectoryRequest request, ServerCallContext context) { - throw new NotImplementedException("TODO: T3.1"); + return _session.ChangeDirectory(request, context.CancellationToken); } public override Task GetLogFiles(Empty empty, ServerCallContext context) { - throw new NotImplementedException("TODO: T3.1"); + return _session.GetLogFiles(empty, context.CancellationToken); } public override Task AnalyzeAll(AnalyzeAllRequest request, ServerCallContext context) { - throw new NotImplementedException("TODO: T3.1"); + return _session.AnalyzeAll(request, context.CancellationToken); } public override Task AnalyzeFiles(AnalyzeFilesRequest request, ServerCallContext context) { - throw new NotImplementedException("TODO: T3.1"); + return _session.AnalyzeFiles(request, context.CancellationToken); } public override async Task GetAnalysisResult(GetAnalysisResultRequest request, IServerStreamWriter responseStream, ServerCallContext context) { - throw new NotImplementedException("TODO: T3.1"); + var responses = _session.GetAnalysisResult(request, context.CancellationToken); + + foreach (var response in responses) + { + await responseStream.WriteAsync(response); + } } } } diff --git a/src/LogAnalyzerRpc/GrpcLogEntryVisitor.cs b/src/LogAnalyzerRpc/GrpcLogEntryVisitor.cs index eb69232..2b31f06 100644 --- a/src/LogAnalyzerRpc/GrpcLogEntryVisitor.cs +++ b/src/LogAnalyzerRpc/GrpcLogEntryVisitor.cs @@ -30,12 +30,38 @@ public LogEntryMessage Visit(CallLogEntry entry) public LogEntryMessage Visit(RequestLogEntry entry) { - throw new NotImplementedException("TODO: T3.1"); + return new LogEntryMessage() + { + RequestLogEntry=new RequestLogEntryMessage + { + LineNo = entry.LineNo, + Timestamp = Timestamp.FromDateTimeOffset(entry.Timestamp), + PodName = entry.PodName, + Severity = GrpcTypeConverter.ConvertToGrpc(entry.Severity), + EventType = GrpcTypeConverter.ConvertToGrpc(entry.EventType), + RequestId = entry.RequestId, + Method = entry.Method, + Path = entry.Path, + StatusCode = entry.StatusCode + } + }; } public LogEntryMessage Visit(InternalLogEntry entry) { - throw new NotImplementedException("TODO: T3.1"); + return new LogEntryMessage() + { + InternalLogEntry = new InternalLogEntryMessage + { + LineNo = entry.LineNo, + Timestamp = Timestamp.FromDateTimeOffset(entry.Timestamp), + PodName = entry.PodName, + Severity = GrpcTypeConverter.ConvertToGrpc(entry.Severity), + EventType = GrpcTypeConverter.ConvertToGrpc(entry.EventType), + ExceptionName = entry.ExceptionName, + ExceptionMessage = entry.ExceptionMessage + } + }; } } } diff --git a/src/LogAnalyzerRpc/GrpcTypeConverter.cs b/src/LogAnalyzerRpc/GrpcTypeConverter.cs index 029122e..5f53290 100644 --- a/src/LogAnalyzerRpc/GrpcTypeConverter.cs +++ b/src/LogAnalyzerRpc/GrpcTypeConverter.cs @@ -1,4 +1,5 @@ -using Google.Protobuf.WellKnownTypes; +using System.Data; +using Google.Protobuf.WellKnownTypes; using LogAnalyzer; using LogAnalyzerRpc.Protos; using LogParser.Models; @@ -20,12 +21,24 @@ public static AnalysisStateEnum ConvertToGrpc(AnalysisState state) public static LogSeverityEnum ConvertToGrpc(LogSeverity severity) { - throw new NotImplementedException("TODO: T3.1"); + return severity switch + { + LogSeverity.Error=>LogSeverityEnum.Error, + LogSeverity.Info=>LogSeverityEnum.Info, + LogSeverity.Warning=>LogSeverityEnum.Warning, + _ => throw new ArgumentOutOfRangeException(nameof(severity), severity, null) + }; } public static LogEventTypeEnum ConvertToGrpc(LogEventType eventType) { - throw new NotImplementedException("TODO: T3.1"); + return eventType switch + { + LogEventType.Call => LogEventTypeEnum.Call, + LogEventType.Request => LogEventTypeEnum.Request, + LogEventType.Internal => LogEventTypeEnum.Internal, + _ => throw new ArgumentOutOfRangeException(nameof(eventType), eventType, null) + }; } public static LogEntryMessage ConvertToGrpc(LogEntry entry) @@ -46,12 +59,24 @@ public static AnalysisState ConvertFromGrpc(AnalysisStateEnum state) public static LogSeverity ConvertFromGrpc(LogSeverityEnum severity) { - throw new NotImplementedException("TODO: T3.1"); + return severity switch + { + LogSeverityEnum.Info => LogSeverity.Info, + LogSeverityEnum.Warning => LogSeverity.Warning, + LogSeverityEnum.Error => LogSeverity.Error, + _ => throw new ArgumentOutOfRangeException(nameof(severity), severity, null) + }; } public static LogEventType ConvertFromGrpc(LogEventTypeEnum eventType) { - throw new NotImplementedException("TODO: T3.1"); + return eventType switch + { + LogEventTypeEnum.Call => LogEventType.Call, + LogEventTypeEnum.Request => LogEventType.Request, + LogEventTypeEnum.Internal => LogEventType.Internal, + _ => throw new ArgumentOutOfRangeException(nameof(eventType), eventType, null) + }; } public static LogEntry ConvertFromGrpc(LogEntryMessage entryMessage) @@ -67,8 +92,24 @@ public static LogEntry ConvertFromGrpc(LogEntryMessage entryMessage) TargetService: entryMessage.CallLogEntry.TargetService, DurationMs: entryMessage.CallLogEntry.DurationMs ), - LogEntryMessage.EntryOneofCase.RequestLogEntry => throw new NotImplementedException("TODO: T3.1"), - LogEntryMessage.EntryOneofCase.InternalLogEntry => throw new NotImplementedException("TODO: T3.1"), + LogEntryMessage.EntryOneofCase.RequestLogEntry => new RequestLogEntry( + LineNo: entryMessage.RequestLogEntry.LineNo, + Timestamp: entryMessage.RequestLogEntry.Timestamp.ToDateTimeOffset(), + PodName: entryMessage.RequestLogEntry.PodName, + Severity: ConvertFromGrpc(entryMessage.RequestLogEntry.Severity), + RequestId: entryMessage.RequestLogEntry.RequestId, + Method: entryMessage.RequestLogEntry.Method, + Path: entryMessage.RequestLogEntry.Path, + StatusCode: entryMessage.RequestLogEntry.StatusCode + ), + LogEntryMessage.EntryOneofCase.InternalLogEntry => new InternalLogEntry( + LineNo: entryMessage.InternalLogEntry.LineNo, + Timestamp: entryMessage.InternalLogEntry.Timestamp.ToDateTimeOffset(), + PodName: entryMessage.InternalLogEntry.PodName, + Severity: ConvertFromGrpc(entryMessage.InternalLogEntry.Severity), + ExceptionName: entryMessage.InternalLogEntry.ExceptionName, + ExceptionMessage: entryMessage.InternalLogEntry.ExceptionMessage + ), _ => throw new ArgumentException($"Unknown entry type: {entryMessage.EntryCase}", nameof(entryMessage)) }; } diff --git a/src/LogAnalyzerRpc/LogAnalyzerRpc.csproj b/src/LogAnalyzerRpc/LogAnalyzerRpc.csproj index 0d13f38..2aa0d21 100644 --- a/src/LogAnalyzerRpc/LogAnalyzerRpc.csproj +++ b/src/LogAnalyzerRpc/LogAnalyzerRpc.csproj @@ -6,10 +6,14 @@ enable - - - - + + + + + + runtime; build; native; contentfiles; analyzers; buildtransitive + all + diff --git a/src/LogParser/Models/LogEntries.cs b/src/LogParser/Models/LogEntries.cs index 69edbc0..e4e9bbc 100644 --- a/src/LogParser/Models/LogEntries.cs +++ b/src/LogParser/Models/LogEntries.cs @@ -54,7 +54,7 @@ public sealed record RequestLogEntry( { public override TResult Accept(ILogEntryVisitor visitor) { - throw new NotImplementedException("TODO: T1.2"); + return visitor.Visit(this); } } @@ -69,7 +69,7 @@ public sealed record InternalLogEntry( { public override TResult Accept(ILogEntryVisitor visitor) { - throw new NotImplementedException("TODO: T1.2"); + return visitor.Visit(this); } } diff --git a/src/LogParser/Parser/LineParser.cs b/src/LogParser/Parser/LineParser.cs index 0475f6b..c20338e 100644 --- a/src/LogParser/Parser/LineParser.cs +++ b/src/LogParser/Parser/LineParser.cs @@ -1,4 +1,5 @@ -using LogParser.Models; +using CsvHelper; +using LogParser.Models; using System.Text.Json; using System.Text.Json.Serialization; @@ -16,8 +17,8 @@ public static LogEntry ParseLine(LogRecord logRecord) return eventElement.GetString() switch { "call" => LineParser.CreateCall(logRecord), - "request" => throw new NotImplementedException("TODO: T1.2"), - "internal" => throw new NotImplementedException("TODO: T1.2"), + "request" => LineParser.CreateRequest(logRecord), + "internal" => LineParser.CreateInternal(logRecord), _ => throw new FormatException($"Unknown event type: {eventElement.GetString()} in log message: {logRecord.Message}") }; } @@ -50,12 +51,35 @@ private static LogEntry CreateCall(LogRecord logRecord) private static LogEntry CreateRequest(LogRecord logRecord) { - throw new NotImplementedException("TODO: T1.2"); + var requestMessage = JsonSerializer.Deserialize(logRecord.Message, options) + ?? throw new FormatException($"Failed to deserialize request message: {logRecord.Message}"); + return new RequestLogEntry( + LineNo: logRecord.LineNo, + Timestamp: DateTimeOffset.Parse(logRecord.Timestamp), + PodName: logRecord.PodName, + Severity: ParseSeverity(requestMessage.Severity), + RequestId: requestMessage.RequestId, + Method: requestMessage.Method, + Path:requestMessage.Path, + StatusCode:requestMessage.StatusCode + ); } private static LogEntry CreateInternal(LogRecord logRecord) { - throw new NotImplementedException("TODO: T1.2"); + var internalMessage = JsonSerializer.Deserialize(logRecord.Message, options) + ?? throw new FormatException($"Failed to deserialize internal message: {logRecord.Message}"); + var index=internalMessage.Exception.IndexOf(": "); + var exceptionMessage=internalMessage.Exception.Substring(index+2); + var exceptionName=internalMessage.Exception.Substring(0,index); + return new InternalLogEntry( + LineNo: logRecord.LineNo, + Timestamp: DateTimeOffset.Parse(logRecord.Timestamp), + PodName: logRecord.PodName, + Severity: ParseSeverity(internalMessage.Severity), + ExceptionMessage:exceptionMessage, + ExceptionName:exceptionName + ); } private static LogSeverity ParseSeverity(string severity) @@ -77,11 +101,16 @@ private record CallMessage( ); private record RequestMessage( - // TODO: T1.2 + [property: JsonRequired] string Severity, + [property: JsonRequired] string RequestId, + [property: JsonRequired] string Method, + [property: JsonRequired] string Path, + [property: JsonRequired] int StatusCode ); private record InternalMessage( - // TODO: T1.2 + [property: JsonRequired] string Severity, + [property: JsonRequired] string Exception ); } } diff --git a/src/LogParser/Visitors/KeyValueVisitor.cs b/src/LogParser/Visitors/KeyValueVisitor.cs index e5ceba2..c0ae12e 100644 --- a/src/LogParser/Visitors/KeyValueVisitor.cs +++ b/src/LogParser/Visitors/KeyValueVisitor.cs @@ -26,12 +26,31 @@ public Dictionary Visit(CallLogEntry entry) public Dictionary Visit(RequestLogEntry entry) { - throw new NotImplementedException("TODO: T1.3"); + return new Dictionary + { + ["LineNo"] = entry.LineNo.ToString(), + ["Timestamp"] = entry.Timestamp.ToString("O"), + ["PodName"] = entry.PodName, + ["Severity"] = entry.Severity.ToString(), + ["EventType"] = entry.EventType.ToString(), + ["RequestId"] = entry.RequestId, + ["Method"] = entry.Method, + ["Path"] = entry.Path, + ["StatusCode"] = entry.StatusCode.ToString(), + }; } public Dictionary Visit(InternalLogEntry entry) { - throw new NotImplementedException("TODO: T1.3"); + return new Dictionary{ + ["LineNo"] = entry.LineNo.ToString(), + ["Timestamp"] = entry.Timestamp.ToString("O"), + ["PodName"] = entry.PodName, + ["Severity"] = entry.Severity.ToString(), + ["EventType"] = entry.EventType.ToString(), + ["ExceptionName"] = entry.ExceptionName, + ["ExceptionMessage"] = entry.ExceptionMessage, + }; } } } diff --git a/src/RemoteCli/Program.cs b/src/RemoteCli/Program.cs index de0ac99..70b1079 100644 --- a/src/RemoteCli/Program.cs +++ b/src/RemoteCli/Program.cs @@ -116,32 +116,151 @@ 6. Exit. private static async Task ShowLogFiles(LogAnalyzerAgentServiceClient client) { - throw new NotImplementedException("TODO: T3.2"); + var response = await client.GetLogFilesAsync(new Empty()); + if (!response.Status.Success) + { + Console.WriteLine($"Error: {response.Status.Message}"); + return; + } + var files=response.FileNames; + if (files.Count == 0) + { + Console.WriteLine("No log files found in the current directory."); + return; + } + int index = 1; + foreach (var file in files) + { + Console.WriteLine($"{index}.{file}"); + index++; + } } private static int ReadDegreeOfParallelism() { - throw new NotImplementedException("TODO: T3.2"); + int degree = 0; + while (true) + { + Console.WriteLine("Please input degree of parallelism:"); + var degreeStr = Console.ReadLine(); + if (degreeStr is null) return 0; + if (!int.TryParse(degreeStr, out degree)) + { + Console.WriteLine("Invalid number, please try again."); + continue; + } + break; + } + return degree; } private static List ReadFileNames() { - throw new NotImplementedException("TODO: T3.2"); - } + while (true) + { + Console.WriteLine("Please input log files to analyze (separated by comma):"); + var str = Console.ReadLine(); + if (str is null) return new List(); + var selectedFiles = str.Split(',', StringSplitOptions.RemoveEmptyEntries) + .Select(f => f.Trim()) + .ToList(); + if (selectedFiles.Count == 0) + { + Console.WriteLine("No files specified, please try again."); + continue; + } + return selectedFiles; + } + } private static async Task AnalyzeFiles(LogAnalyzerAgentServiceClient client) { - throw new NotImplementedException("TODO: T3.2"); + var files = ReadFileNames(); + if (files.Count == 0) return; + int degree = ReadDegreeOfParallelism(); + if (degree == 0) return; + var request = new AnalyzeFilesRequest { DegreeOfParallelism = degree }; + request.FileNames.AddRange(files); + var response = await client.AnalyzeFilesAsync(request); + if (!response.Status.Success) + { + Console.WriteLine($"Error: {response.Status.Message}"); + } + else + { + Console.WriteLine("Analysis started successfully."); + } } private static async Task AnalyzeAll(LogAnalyzerAgentServiceClient client) { - throw new NotImplementedException("TODO: T3.2"); + int degree = ReadDegreeOfParallelism(); + if(degree==0){return;} + var request = new AnalyzeAllRequest { DegreeOfParallelism = degree }; + var response = await client.AnalyzeAllAsync(request); + if (!response.Status.Success) + { + Console.WriteLine($"Error: {response.Status.Message}"); + } + else + { + Console.WriteLine("Analysis started successfully."); + } } private static async Task GetAnalysisResult(LogAnalyzerAgentServiceClient client) { - throw new NotImplementedException("TODO: T3.2"); - } - } + Console.WriteLine("Please input log file name:"); + var fileName = Console.ReadLine()?.Trim(); + if (string.IsNullOrEmpty(fileName)) + { + Console.WriteLine("Invalid file name."); + return; + } + var request = new GetAnalysisResultRequest { FileName = fileName }; + using var call = client.GetAnalysisResult(request); + var visitor = new KeyValueVisitor(); + try + { + await foreach (var response in call.ResponseStream.ReadAllAsync()) + { + if (!response.Status.Success) + { + Console.WriteLine($"Error: {response.Status.Message}"); + return; + } + switch (response.PayloadCase) + { + case GetAnalysisResultResponse.PayloadOneofCase.Header: + switch (response.Header.State) + { + case AnalysisStateEnum.NotAnalyzed: + Console.WriteLine("Not analyzed."); + break; + case AnalysisStateEnum.Failed: + Console.WriteLine($"Failed: {response.Header.ErrorMessage}"); + break; + case AnalysisStateEnum.Succeeded: + break; + } + break; + + case GetAnalysisResultResponse.PayloadOneofCase.LogEntry: + var entry = GrpcTypeConverter.ConvertFromGrpc(response.LogEntry); + var logDict = visitor.Dump(entry); + foreach (var kvp in logDict) + { + Console.WriteLine($"{kvp.Key}: {kvp.Value}"); + } + Console.WriteLine("------------------------"); + break; + } + } + } + catch (RpcException ex) + { + Console.WriteLine($"RPC Error: {ex.Status.Detail}"); + } + } + } }